2024-02-10T05:25:17.695 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T05:25:17.695 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --verbose --archive /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078 --owner scheduled_yuriw@teuthology --name yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078/orig.config.yaml 2024-02-10T05:25:17.730 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T05:25:17.816 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078 branch: main description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: yweinste@redhat.com first_in_suite: false job_id: '7555078' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 1610 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGQ/7PSiIHjpmgrRBmPojrxiYa0EX+4kultoEOa+dZsiOf8UfXHSVMYjDpXTN/mO09RFJpV1Qcu2JcxyfLp5Kok= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLzI7ZOpv/aTGxmsFdFemq4T22bYmOkLj0RD4x7ud+DYc1lBem6EmfdN9q5jEKT1ZWg0cUAsfNHs6o2zwxjn4wE= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:44:16 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-02-10T05:25:17.816 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-10T05:25:17.817 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-10T05:25:17.817 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T05:25:17.818 INFO:teuthology.task.internal:Checking packages... 2024-02-10T05:25:17.843 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-10T05:25:17.843 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T05:25:17.843 INFO:teuthology.packaging:ref: None 2024-02-10T05:25:17.843 INFO:teuthology.packaging:tag: None 2024-02-10T05:25:17.844 INFO:teuthology.packaging:branch: main 2024-02-10T05:25:17.844 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:25:17.844 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-10T05:25:18.117 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-10T05:25:18.118 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T05:25:18.125 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T05:25:18.125 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T05:25:18.138 INFO:teuthology.task.internal:Saving configuration 2024-02-10T05:25:18.152 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T05:25:18.158 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:25:18.183 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:20:40.379279', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGQ/7PSiIHjpmgrRBmPojrxiYa0EX+4kultoEOa+dZsiOf8UfXHSVMYjDpXTN/mO09RFJpV1Qcu2JcxyfLp5Kok='} 2024-02-10T05:25:18.204 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:20:40.381044', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLzI7ZOpv/aTGxmsFdFemq4T22bYmOkLj0RD4x7ud+DYc1lBem6EmfdN9q5jEKT1ZWg0cUAsfNHs6o2zwxjn4wE='} 2024-02-10T05:25:18.204 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T05:25:18.211 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-02-10T05:25:18.211 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-02-10T05:25:18.211 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T05:25:18.281 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9f463971f0>, signals=[15]) 2024-02-10T05:25:18.282 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T05:25:18.290 INFO:teuthology.task.internal:Opening connections... 2024-02-10T05:25:18.290 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T05:25:18.293 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:25:18.396 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-02-10T05:25:18.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:25:18.485 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T05:25:18.492 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-02-10T05:25:18.509 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-02-10T05:25:18.509 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-02-10T05:25:18.565 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-02-10T05:25:18.565 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-02-10T05:25:18.565 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-02-10T05:25:18.565 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:25:18.565 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-02-10T05:25:18.566 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:25:18.566 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:25:18.566 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-02-10T05:25:18.566 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:25:18.566 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:25:18.566 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:25:18.566 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:25:18.566 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:25:18.566 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:25:18.567 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-02-10T05:25:18.591 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-02-10T05:25:18.608 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-02-10T05:25:18.609 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-02-10T05:25:18.665 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-02-10T05:25:18.665 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2024-02-10T05:25:18.665 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-02-10T05:25:18.665 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:25:18.665 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2024-02-10T05:25:18.665 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:25:18.665 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:25:18.666 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-02-10T05:25:18.666 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:25:18.666 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:25:18.666 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:25:18.666 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:25:18.666 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:25:18.666 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:25:18.666 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-02-10T05:25:18.690 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T05:25:18.699 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T05:25:18.704 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T05:25:18.705 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:25:18.707 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:25:18.723 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T05:25:18.729 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T05:25:18.729 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:25:18.764 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:25:18.859 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T05:25:18.916 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T05:25:18.931 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T05:25:18.932 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T05:25:18.932 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:25:18.932 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:25:18.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:25:18.949 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-02-10T05:25:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:25:18.950 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-02-10T05:25:19.005 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:25:19.006 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.14.0-361.el9.x86_64 2024-02-10T05:25:19.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2024-02-10T05:25:19.007 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:25:19.008 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-361.el9.x86_64 2024-02-10T05:25:19.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-02-10T05:25:21.163 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-02-10T05:25:21.254 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-10T05:25:22.680 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-02-10T05:25:22.913 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-02-10T05:25:23.948 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-02-10T05:25:24.454 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.0 MB 00:01 2024-02-10T05:25:26.638 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-02-10T05:25:27.132 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-02-10T05:25:31.697 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 5.9 MB 00:01 2024-02-10T05:25:32.531 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 5.9 MB 00:01 2024-02-10T05:25:33.147 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 15 kB 00:00 2024-02-10T05:25:33.931 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 15 kB 00:00 2024-02-10T05:25:35.758 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-02-10T05:25:40.506 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 2.8 MB/s | 20 MB 00:07 2024-02-10T05:25:40.874 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-02-10T05:25:41.086 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:01 ago on Sat 10 Feb 2024 05:25:40 AM UTC. 2024-02-10T05:25:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:25:43.223 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:25:43.326 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-10T05:25:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-10T05:25:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-10T05:25:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-10T05:25:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-10T05:25:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:25:43.337 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-02-10T05:25:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:25:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:25:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:25:43.338 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:25:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-10T05:25:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-10T05:25:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-02-10T05:25:43.338 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:25:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 99 M 2024-02-10T05:25:43.338 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 122 M 2024-02-10T05:25:43.339 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-10T05:25:44.393 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 5.7 MB/s | 6.0 MB 00:01 2024-02-10T05:25:45.243 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 11 MB/s | 21 MB 00:01 2024-02-10T05:25:45.673 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 5.6 kB/s | 1.7 kB 00:00 2024-02-10T05:25:46.443 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 16 MB/s | 33 MB 00:02 2024-02-10T05:25:47.051 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 11 MB/s | 39 MB 00:03 2024-02-10T05:25:47.055 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:25:47.055 INFO:teuthology.orchestra.run.smithi028.stdout:Total 27 MB/s | 99 MB 00:03 2024-02-10T05:25:47.056 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-10T05:25:47.519 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-10T05:25:47.520 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-10T05:25:47.997 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:25:47.997 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:25:48.092 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-02-10T05:25:48.104 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-10T05:25:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-02-10T05:25:48.104 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-10T05:25:48.104 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-02-10T05:25:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:25:48.104 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-02-10T05:25:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:25:48.105 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:25:48.105 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:25:48.105 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:25:48.105 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-02-10T05:25:48.105 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-02-10T05:25:48.105 INFO:teuthology.orchestra.run.smithi192.stdout:Install 4 Packages 2024-02-10T05:25:48.105 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:25:48.105 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 99 M 2024-02-10T05:25:48.106 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 122 M 2024-02-10T05:25:48.106 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-02-10T05:25:48.600 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-10T05:25:48.600 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-10T05:25:49.016 INFO:teuthology.orchestra.run.smithi192.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 6.6 MB/s | 6.0 MB 00:00 2024-02-10T05:25:49.966 INFO:teuthology.orchestra.run.smithi192.stdout:(2/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-02-10T05:25:50.186 INFO:teuthology.orchestra.run.smithi192.stdout:(3/4): kernel-core-5.14.0-419.el9.x86_64.rpm 10 MB/s | 21 MB 00:02 2024-02-10T05:25:51.129 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-10T05:25:51.191 INFO:teuthology.orchestra.run.smithi192.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 15 MB/s | 33 MB 00:02 2024-02-10T05:25:51.195 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:25:51.195 INFO:teuthology.orchestra.run.smithi192.stdout:Total 32 MB/s | 99 MB 00:03 2024-02-10T05:25:51.195 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-02-10T05:25:51.578 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-02-10T05:25:51.578 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-02-10T05:25:51.805 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:25:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:25:52.626 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-02-10T05:25:52.626 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-02-10T05:25:54.845 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:25:55.043 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-02-10T05:25:55.045 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:25:55.690 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:25:55.828 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:25:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:25:58.754 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:26:00.109 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:26:00.217 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:26:03.453 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:26:03.641 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:26:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:26:08.366 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:26:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:26:36.164 INFO:teuthology.orchestra.run.smithi028.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:26:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:26:36.236 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:26:36.678 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:26:36.678 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:26:36.678 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:26:36.678 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:26:37.118 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:26:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:26:37.119 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-10T05:26:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:26:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:26:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:26:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:26:37.119 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:26:37.119 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-10T05:26:37.616 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:26:37.616 INFO:teuthology.orchestra.run.smithi192.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:26:37.616 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:26:37.620 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:26:37.686 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:26:37.758 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:26:37.759 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:26:37.759 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:26:37.759 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:26:37.759 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:26:37.759 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-10T05:26:37.760 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:26:37.760 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:26:37.872 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:26:37.872 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:26:37.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-02-10T05:26:37.925 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 68.el9 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8264254 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:26:37.926 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-02-10T05:26:37.927 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-02-10T05:26:37.927 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:26:37.927 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-02-10T05:26:37.927 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-02-10T05:26:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:26:37.927 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:26:37.927 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:26:37.927 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:26:37.927 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-02-10T05:26:37.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:26:37.927 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:26:37.928 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:26:37.928 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:26:37.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:26:38.186 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:26:38.187 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:26:38.187 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:26:38.187 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:26:38.531 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-02-10T05:26:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:26:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:26:38.655 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-02-10T05:26:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:26:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:26:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:26:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:26:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:26:38.656 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-02-10T05:26:39.190 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:26:39.320 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:26:39.320 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:26:39.320 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:26:39.320 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:26:39.320 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:26:39.320 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-10T05:26:39.320 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:26:39.321 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:26:39.428 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:26:39.429 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:26:39.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo rpm -qi grub2-tools 2024-02-10T05:26:39.463 INFO:teuthology.orchestra.run.smithi192.stdout:Name : grub2-tools 2024-02-10T05:26:39.463 INFO:teuthology.orchestra.run.smithi192.stdout:Epoch : 1 2024-02-10T05:26:39.463 INFO:teuthology.orchestra.run.smithi192.stdout:Version : 2.06 2024-02-10T05:26:39.463 INFO:teuthology.orchestra.run.smithi192.stdout:Release : 68.el9 2024-02-10T05:26:39.463 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: x86_64 2024-02-10T05:26:39.463 INFO:teuthology.orchestra.run.smithi192.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:26:39.463 INFO:teuthology.orchestra.run.smithi192.stdout:Group : Unspecified 2024-02-10T05:26:39.463 INFO:teuthology.orchestra.run.smithi192.stdout:Size : 8264254 2024-02-10T05:26:39.463 INFO:teuthology.orchestra.run.smithi192.stdout:License : GPLv3+ 2024-02-10T05:26:39.463 INFO:teuthology.orchestra.run.smithi192.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:Packager : builder@centos.org 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:Vendor : CentOS 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:Summary : Support tools for GRUB. 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:Description : 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:hardware devices. 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:26:39.464 INFO:teuthology.orchestra.run.smithi192.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:26:39.465 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:26:39.465 INFO:teuthology.task.kernel:Updating grub on smithi192 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:26:39.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:26:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:26:40.067 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-02-10T05:26:40.077 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-02-10T05:26:40.078 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2024-02-10T05:26:40.093 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.YpeU8NRAeG 2024-02-10T05:26:40.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YpeU8NRAeG 2024-02-10T05:26:40.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.YpeU8NRAeG 2024-02-10T05:26:40.320 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.YpeU8NRAeG is 6KB 2024-02-10T05:26:40.370 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.YpeU8NRAeG 2024-02-10T05:26:40.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:26:40.453 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:26:40.453 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:26:40.453 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:26:40.454 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:26:40.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:26:40.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-02-10T05:26:41.392 INFO:teuthology.orchestra.run.smithi192.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:26:41.427 INFO:teuthology.orchestra.run.smithi192.stderr:done 2024-02-10T05:26:41.429 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-02-10T05:26:41.444 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.vK5RFmM6R9 2024-02-10T05:26:41.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vK5RFmM6R9 2024-02-10T05:26:41.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.vK5RFmM6R9 2024-02-10T05:26:41.661 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.vK5RFmM6R9 is 6KB 2024-02-10T05:26:41.709 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.vK5RFmM6R9 2024-02-10T05:26:41.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:26:41.788 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:26:41.788 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:26:41.789 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:26:41.789 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:26:41.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:26:41.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2024-02-10T05:27:10.636 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-10T05:27:10.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-10T05:27:10.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:27:11.948 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-10T05:27:11.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-02-10T05:27:11.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:28:10.697 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T05:28:11.951 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T05:28:19.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-10T05:28:19.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:28:20.023 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-02-10T05:28:20.428 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-10T05:28:20.428 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:28:20.428 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-02-10T05:28:20.444 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:28:20.444 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:28:20.445 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:28:20.445 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:28:20.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-02-10T05:28:20.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:28:21.317 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-02-10T05:28:21.445 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-10T05:28:21.446 DEBUG:teuthology.orchestra.run.smithi028:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:28:21.536 INFO:teuthology.orchestra.run.smithi028.stdout:ttyS1 2024-02-10T05:28:21.568 DEBUG:teuthology.parallel:result is None 2024-02-10T05:28:21.756 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-02-10T05:28:21.756 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:28:21.757 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-02-10T05:28:21.771 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:28:21.771 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:28:21.771 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:28:21.772 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:28:22.772 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-10T05:28:22.772 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:28:22.830 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2024-02-10T05:28:22.874 DEBUG:teuthology.parallel:result is None 2024-02-10T05:28:22.874 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T05:28:22.882 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T05:28:22.882 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:28:22.884 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:28:22.899 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T05:28:22.907 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T05:28:22.915 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T05:28:22.915 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:28:22.942 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:28:22.975 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T05:28:22.982 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T05:28:22.982 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:28:23.014 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:28:23.047 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:28:23.051 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:28:23.061 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:28:23.064 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:28:23.066 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T05:28:23.073 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T05:28:23.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:28:23.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:28:23.135 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T05:28:23.145 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T05:28:23.145 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:28:23.178 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:28:23.193 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:28:23.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:28:23.328 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:28:23.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:28:23.456 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-10T05:28:23.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:28:23.525 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:28:23.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:28:23.614 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:28:23.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:28:23.744 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-10T05:28:23.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:28:23.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-02-10T05:28:23.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-02-10T05:28:23.867 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:28:23.901 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:28:24.346 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T05:28:24.355 INFO:teuthology.task.internal:Starting timer... 2024-02-10T05:28:24.356 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T05:28:24.366 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T05:28:24.374 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-10T05:28:24.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-02-10T05:28:24.434 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-02-10T05:28:24.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-02-10T05:28:24.499 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-02-10T05:28:24.500 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T05:28:24.500 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-02-10T05:28:24.546 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-02-10T05:28:24.546 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-02-10T05:28:24.567 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-02-10T05:28:24.567 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-02-10T05:28:24.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:28:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:28:24.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:28:24.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:28:24.638 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T05:28:24.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:28:24.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:28:24.777 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T05:28:24.787 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T05:28:24.805 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T05:28:24.806 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T05:32:40.499 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-02-10T05:32:40.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-10T05:32:40.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:32:40.596 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-02-10T05:32:40.679 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-10T05:32:40.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-02-10T05:32:40.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:32:40.772 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-02-10T05:32:40.856 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-02-10T05:32:40.857 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T05:32:40.868 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T05:32:40.868 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T05:32:40.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:32:40.871 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T05:32:40.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:32:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T05:32:40.918 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T05:32:40.929 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T05:32:40.947 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T05:32:40.948 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-02-10T05:32:40.962 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-02-10T05:32:40.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-02-10T05:32:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T05:32:40.991 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-02-10T05:32:40.995 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T05:32:41.008 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T05:32:41.024 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T05:32:41.046 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:32:41.051 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:32:41.051 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-02-10T05:32:41.051 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:32:41.051 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:32:41.051 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:32:41.051 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:32:41.077 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:32:41.080 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:32:41.080 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-02-10T05:32:41.081 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:32:41.081 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:32:41.081 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:32:41.081 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:32:41.082 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T05:32:41.193 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T05:32:41.193 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-02-10T05:32:41.193 INFO:tasks.cephadm:Cluster fsid is cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:32:41.194 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T05:32:41.194 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-10T05:32:41.194 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi028': '172.21.15.28', 'mon.smithi192': '172.21.15.192'} 2024-02-10T05:32:41.194 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T05:32:41.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-02-10T05:32:41.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2024-02-10T05:32:41.250 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-02-10T05:32:41.250 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T05:32:41.511 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 10 05:32 /home/ubuntu/cephtest/cephadm 2024-02-10T05:32:41.511 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T05:32:41.792 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 10 05:32 /home/ubuntu/cephtest/cephadm 2024-02-10T05:32:41.793 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T05:32:41.808 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T05:32:41.833 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-02-10T05:32:41.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-10T05:32:41.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-10T05:32:42.656 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-10T05:32:42.758 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-10T05:33:02.298 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:33:02.298 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-10T05:33:02.298 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-10T05:33:02.299 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-02-10T05:33:02.299 INFO:teuthology.orchestra.run.smithi028.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-10T05:33:02.299 INFO:teuthology.orchestra.run.smithi028.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-10T05:33:02.299 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-10T05:33:02.299 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:33:04.296 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-02-10T05:33:04.296 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-10T05:33:04.296 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-10T05:33:04.296 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2024-02-10T05:33:04.297 INFO:teuthology.orchestra.run.smithi192.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-10T05:33:04.297 INFO:teuthology.orchestra.run.smithi192.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-10T05:33:04.297 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-02-10T05:33:04.297 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-02-10T05:33:04.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-02-10T05:33:04.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2024-02-10T05:33:04.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-02-10T05:33:04.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2024-02-10T05:33:04.446 INFO:tasks.cephadm:Writing seed config... 2024-02-10T05:33:04.447 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T05:33:04.447 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T05:33:04.447 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T05:33:04.447 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T05:33:04.447 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T05:33:04.447 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T05:33:04.448 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T05:33:04.448 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-10T05:33:04.448 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-10T05:33:04.448 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T05:33:04.476 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = cf1ef3cc-c7d5-11ee-95b8-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-10T05:33:04.476 DEBUG:teuthology.orchestra.run.smithi028:mon.smithi028> sudo journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi028.service 2024-02-10T05:33:04.519 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T05:33:04.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.28 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:33:04.721 INFO:teuthology.orchestra.run.smithi028.stderr:-------------------------------------------------------------------------------- 2024-02-10T05:33:04.721 INFO:teuthology.orchestra.run.smithi028.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'cf1ef3cc-c7d5-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.28', '--skip-admin-label'] 2024-02-10T05:33:04.745 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: 4.9.3-dev 2024-02-10T05:33:04.747 INFO:teuthology.orchestra.run.smithi028.stderr:Verifying podman|docker is present... 2024-02-10T05:33:04.769 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: 4.9.3-dev 2024-02-10T05:33:04.772 INFO:teuthology.orchestra.run.smithi028.stderr:Verifying lvm2 is present... 2024-02-10T05:33:04.772 INFO:teuthology.orchestra.run.smithi028.stderr:Verifying time synchronization is in place... 2024-02-10T05:33:04.778 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T05:33:04.784 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: inactive 2024-02-10T05:33:04.791 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: enabled 2024-02-10T05:33:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: active 2024-02-10T05:33:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:Unit chronyd.service is enabled and running 2024-02-10T05:33:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:Repeating the final host check... 2024-02-10T05:33:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: 4.9.3-dev 2024-02-10T05:33:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:podman (/bin/podman) version 4.9.3 is present 2024-02-10T05:33:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl is present 2024-02-10T05:33:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:lvcreate is present 2024-02-10T05:33:04.826 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T05:33:04.832 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: inactive 2024-02-10T05:33:04.838 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: enabled 2024-02-10T05:33:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: active 2024-02-10T05:33:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:Unit chronyd.service is enabled and running 2024-02-10T05:33:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:Host looks OK 2024-02-10T05:33:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:Cluster fsid: cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:33:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:Acquiring lock 140678232468880 on /run/cephadm/cf1ef3cc-c7d5-11ee-95b8-87774f69a715.lock 2024-02-10T05:33:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:Lock 140678232468880 acquired on /run/cephadm/cf1ef3cc-c7d5-11ee-95b8-87774f69a715.lock 2024-02-10T05:33:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:Verifying IP 172.21.15.28 port 3300 ... 2024-02-10T05:33:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:Verifying IP 172.21.15.28 port 6789 ... 2024-02-10T05:33:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:Base mon IP is 172.21.15.28, final addrv is [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-02-10T05:33:04.848 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-02-10T05:33:04.848 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-02-10T05:33:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T05:33:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-10T05:33:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-10T05:33:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T05:33:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-10T05:33:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T05:33:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-10T05:33:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-02-10T05:33:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T05:33:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-02-10T05:33:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T05:33:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-10T05:33:04.958 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-02-10T05:33:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Getting image source signatures 2024-02-10T05:33:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-02-10T05:33:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-02-10T05:33:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-02-10T05:33:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-02-10T05:33:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-02-10T05:33:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-10T05:33:05.443 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: Writing manifest to image destination 2024-02-10T05:33:05.595 INFO:teuthology.orchestra.run.smithi028.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-10T05:33:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-10T05:33:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-10T05:33:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:Extracting ceph user uid/gid from container image... 2024-02-10T05:33:06.860 INFO:teuthology.orchestra.run.smithi028.stderr:stat: 167 167 2024-02-10T05:33:07.449 INFO:teuthology.orchestra.run.smithi028.stderr:Creating initial keys... 2024-02-10T05:33:07.807 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph-authtool: AQCTCsdlfJMlLhAAfGiSte4uzGHNSFwV8yWr0Q== 2024-02-10T05:33:08.663 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph-authtool: AQCUCsdlZVvyJBAA/KI925rSMEKagh6Wfs+dUA== 2024-02-10T05:33:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph-authtool: AQCVCsdlFQBwHRAAfIqblhYMTuKzgoXBLBDaMA== 2024-02-10T05:33:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:Creating initial monmap... 2024-02-10T05:33:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T05:33:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-02-10T05:33:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:33:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T05:33:11.079 INFO:teuthology.orchestra.run.smithi028.stderr:monmaptool for smithi028 [v2:172.21.15.28:3300,v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T05:33:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:setting min_mon_release = octopus 2024-02-10T05:33:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/monmaptool: set fsid to cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:33:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T05:33:11.080 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-10T05:33:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:Creating mon... 2024-02-10T05:33:11.976 INFO:teuthology.orchestra.run.smithi028.stderr:create mon.smithi028 on 2024-02-10T05:33:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T05:33:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715.target. 2024-02-10T05:33:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715.target. 2024-02-10T05:33:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Failed to reset failed state of unit ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi028.service: Unit ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi028.service not loaded. 2024-02-10T05:33:12.622 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Created symlink /etc/systemd/system/ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715.target.wants/ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi028.service → /etc/systemd/system/ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@.service. 2024-02-10T05:33:13.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:13 smithi028 podman[24313]: 2024-02-10 05:33:13.179172082 +0000 UTC m=+0.300943571 container start 90f54a1004f6b254d5badc84acd91a46f50e431040403e29cab63784ec8fb1d4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, io.buildah.version=1.19.8, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, distribution-scope=public, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, version=8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., ceph=True) 2024-02-10T05:33:13.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:13 smithi028 ceph-mon[24332]: mkfs cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:33:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: disabled 2024-02-10T05:33:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: inactive 2024-02-10T05:33:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:firewalld.service is not enabled 2024-02-10T05:33:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T05:33:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for mon to start... 2024-02-10T05:33:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for mon... 2024-02-10T05:33:13.509 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:13 smithi028 ceph-mon[24332]: mon.smithi028 is new leader, mons smithi028 in quorum (ranks 0) 2024-02-10T05:33:13.509 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:13 smithi028 bash[24313]: 90f54a1004f6b254d5badc84acd91a46f50e431040403e29cab63784ec8fb1d4 2024-02-10T05:33:13.509 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:13 smithi028 systemd[1]: Started Ceph mon.smithi028 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:33:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: cluster: 2024-02-10T05:33:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: id: cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:33:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-10T05:33:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-02-10T05:33:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: services: 2024-02-10T05:33:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi028 (age 0.552569s) 2024-02-10T05:33:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-10T05:33:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-10T05:33:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-02-10T05:33:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: data: 2024-02-10T05:33:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-10T05:33:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-10T05:33:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-10T05:33:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: pgs: 2024-02-10T05:33:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-02-10T05:33:14.320 INFO:teuthology.orchestra.run.smithi028.stderr:mon is available 2024-02-10T05:33:14.320 INFO:teuthology.orchestra.run.smithi028.stderr:Assimilating anything we can from ceph.conf... 2024-02-10T05:33:14.416 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:14 smithi028 ceph-mon[24332]: mon.smithi028 is new leader, mons smithi028 in quorum (ranks 0) 2024-02-10T05:33:14.416 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:14 smithi028 ceph-mon[24332]: monmap e1: 1 mons at {smithi028=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0]} 2024-02-10T05:33:14.417 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:14 smithi028 ceph-mon[24332]: fsmap 2024-02-10T05:33:14.417 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:14 smithi028 ceph-mon[24332]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T05:33:14.417 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:14 smithi028 ceph-mon[24332]: mgrmap e1: no daemons active 2024-02-10T05:33:14.417 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:14 smithi028 ceph-mon[24332]: from='client.? 172.21.15.28:0/3584443675' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T05:33:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-02-10T05:33:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: [global] 2024-02-10T05:33:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: fsid = cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:33:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-02-10T05:33:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-10T05:33:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-10T05:33:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-10T05:33:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-10T05:33:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-02-10T05:33:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: [mgr] 2024-02-10T05:33:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-10T05:33:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-02-10T05:33:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: [osd] 2024-02-10T05:33:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-10T05:33:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-10T05:33:15.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:15 smithi028 ceph-mon[24332]: from='client.? 172.21.15.28:0/219280565' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T05:33:15.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:15 smithi028 ceph-mon[24332]: from='client.? 172.21.15.28:0/219280565' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T05:33:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:Generating new minimal ceph.conf... 2024-02-10T05:33:16.479 INFO:teuthology.orchestra.run.smithi028.stderr:Restarting the monitor... 2024-02-10T05:33:16.496 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:16 smithi028 ceph-mon[24332]: from='client.? 172.21.15.28:0/3371517444' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:33:16.803 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:16 smithi028 systemd[1]: Stopping Ceph mon.smithi028 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:33:16.803 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:16 smithi028 bash[24551]: Error: no container with name or ID "ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon.smithi028" found: no such container 2024-02-10T05:33:16.803 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:16 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028[24328]: 2024-02-10T05:33:16.628+0000 7fd3d4525700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi028 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:33:16.803 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:16 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028[24328]: 2024-02-10T05:33:16.628+0000 7fd3d4525700 -1 mon.smithi028@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T05:33:17.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:16 smithi028 podman[24563]: 2024-02-10 05:33:16.803337394 +0000 UTC m=+0.270348700 container died 90f54a1004f6b254d5badc84acd91a46f50e431040403e29cab63784ec8fb1d4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vendor=Red Hat, Inc., GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, architecture=x86_64, version=8, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, name=centos-stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_BRANCH=HEAD, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-10T05:33:17.400 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 podman[24563]: 2024-02-10 05:33:17.178756981 +0000 UTC m=+0.645768285 container cleanup 90f54a1004f6b254d5badc84acd91a46f50e431040403e29cab63784ec8fb1d4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, version=8, distribution-scope=public, release=754, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, vcs-type=git, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., io.buildah.version=1.19.8, RELEASE=HEAD) 2024-02-10T05:33:17.400 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 bash[24563]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028 2024-02-10T05:33:17.400 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 bash[24590]: Error: no container with name or ID "ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon.smithi028" found: no such container 2024-02-10T05:33:17.740 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 podman[24577]: 2024-02-10 05:33:17.396242905 +0000 UTC m=+0.589505858 container remove 90f54a1004f6b254d5badc84acd91a46f50e431040403e29cab63784ec8fb1d4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, vcs-type=git, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_CLEAN=True, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, ceph=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, version=8, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8) 2024-02-10T05:33:17.740 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi028.service: Deactivated successfully. 2024-02-10T05:33:17.740 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 systemd[1]: Stopped Ceph mon.smithi028 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:33:17.740 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 systemd[1]: Starting Ceph mon.smithi028 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:33:17.740 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 podman[24662]: 2024-02-10 05:33:17.532094975 +0000 UTC m=+0.018498996 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-02-10T05:33:17.740 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 podman[24662]: 2024-02-10 05:33:17.63015482 +0000 UTC m=+0.116558826 container create a5294cba5afaeca41b2bd8d72367849733d1306c18d2d8b9bde7ece6804f6ad8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, maintainer=Guillaume Abrioux , name=centos-stream, architecture=x86_64, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-10T05:33:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-10T05:33:18.050 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 podman[24662]: 2024-02-10 05:33:17.849205591 +0000 UTC m=+0.335609603 container init a5294cba5afaeca41b2bd8d72367849733d1306c18d2d8b9bde7ece6804f6ad8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, release=754, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , name=centos-stream, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_CLEAN=True) 2024-02-10T05:33:18.050 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 podman[24662]: 2024-02-10 05:33:17.852187397 +0000 UTC m=+0.338591396 container start a5294cba5afaeca41b2bd8d72367849733d1306c18d2d8b9bde7ece6804f6ad8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, RELEASE=HEAD, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, architecture=x86_64, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, release=754, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8) 2024-02-10T05:33:18.050 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:33:18.050 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-10T05:33:18.050 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: pidfile_write: ignore empty --pid-file 2024-02-10T05:33:18.050 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: load: jerasure load: lrc 2024-02-10T05:33:18.050 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: RocksDB version: 6.15.5 2024-02-10T05:33:18.051 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T05:33:18.051 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Compile date Apr 18 2022 2024-02-10T05:33:18.051 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: DB SUMMARY 2024-02-10T05:33:18.051 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: DB Session ID: 1HT6ZWS2F4WQCAJC6R2B 2024-02-10T05:33:18.051 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:33:18.051 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:33:18.051 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-10T05:33:18.051 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi028/store.db dir, Total Num: 1, files: 000008.sst 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi028/store.db: 000010.log size: 82705 ; 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.env: 0x5651b00c6860 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.fs: Posix File System 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.info_log: 0x5651b173dfa0 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.statistics: (nil) 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.use_fsync: 0 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:33:18.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.db_log_dir: 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi028/store.db 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:33:18.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.write_buffer_manager: 0x5651b182e330 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.unordered_write: 0 2024-02-10T05:33:18.054 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.row_cache: None 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.wal_filter: None 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.preserve_deletes: 0 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:33:18.055 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:33:18.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:33:18.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:33:18.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:33:18.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_open_files: -1 2024-02-10T05:33:18.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:33:18.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:33:18.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:33:18.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:33:18.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:33:18.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Compression algorithms supported: 2024-02-10T05:33:18.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:33:18.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: kZSTD supported: 0 2024-02-10T05:33:18.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:33:18.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:33:18.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:33:18.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:33:18.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:33:18.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:33:18.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:33:18.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi028/store.db/MANIFEST-000009 2024-02-10T05:33:18.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-10T05:33:18.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:33:18.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.merge_operator: 2024-02-10T05:33:18.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_filter: None 2024-02-10T05:33:18.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:33:18.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:33:18.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:33:18.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5651b1709d00) 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: index_type: 0 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: data_block_index_type: 0 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: index_shortening: 1 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: hash_index_allow_collision: 1 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: checksum: 1 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: no_block_cache: 0 2024-02-10T05:33:18.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_cache: 0x5651b1774170 2024-02-10T05:33:18.062 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:33:18.062 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_cache_options: 2024-02-10T05:33:18.062 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: capacity : 536870912 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: num_shard_bits : 4 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: strict_capacity_limit : 0 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_cache_compressed: (nil) 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: persistent_cache: (nil) 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_size: 4096 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_size_deviation: 10 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_restart_interval: 16 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: index_block_restart_interval: 1 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: metadata_block_size: 4096 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: partition_filters: 0 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: use_delta_encoding: 1 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T05:33:18.063 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: whole_key_filtering: 1 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: verify_compression: 0 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: format_version: 4 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: enable_index_compression: 1 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_align: 0 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compression: NoCompression 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.num_levels: 7 2024-02-10T05:33:18.064 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:33:18.065 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:33:18.065 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:33:18.065 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:33:18.065 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:33:18.065 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:33:18.065 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:33:18.065 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:33:18.065 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:33:18.065 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:33:18.066 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:33:18.066 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:33:18.066 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:33:18.066 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:33:18.066 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:33:18.066 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:33:18.066 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:33:18.066 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:33:18.066 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:33:18.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:33:18.068 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.arena_block_size: 4194304 2024-02-10T05:33:18.068 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:33:18.068 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:33:18.068 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T05:33:18.068 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:33:18.068 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:33:18.068 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:33:18.068 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:33:18.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:33:18.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:33:18.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:33:18.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:33:18.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:33:18.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:33:18.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:33:18.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.ttl: 2592000 2024-02-10T05:33:18.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:33:18.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:33:18.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi028/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543197878886, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543197879628, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79387, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77651, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71960, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707543197, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "5119927e-0a17-4cd0-b3e2-50f5153f9c1b", "db_session_id": "1HT6ZWS2F4WQCAJC6R2B"}} 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543197879961, "job": 1, "event": "recovery_finished"} 2024-02-10T05:33:18.071 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi028/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5651b175aa80 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: DB pointer 0x5651b176a000 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ** DB Stats ** 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:33:18.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ** Compaction Stats [default] ** 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: L0 2/0 79.32 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Sum 2/0 79.32 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ** Compaction Stats [default] ** 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:33:18.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Cumulative compaction: 0.00 GB write, 25.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Interval compaction: 0.00 GB write, 25.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ** Compaction Stats [default] ** 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: L0 2/0 79.32 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-02-10T05:33:18.074 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Sum 2/0 79.32 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-02-10T05:33:18.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T05:33:18.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:33:18.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ** Compaction Stats [default] ** 2024-02-10T05:33:18.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:33:18.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:33:18.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.4 0.00 0.00 1 0.001 0 0 2024-02-10T05:33:18.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:33:18.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:33:18.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:33:18.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:33:18.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:33:18.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Cumulative compaction: 0.00 GB write, 25.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: starting mon.smithi028 rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi028 fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: mon.smithi028@-1(???) e1 preinit fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: mon.smithi028@-1(???).mds e1 new map 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: mon.smithi028@-1(???).mds e1 print_map 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: e1 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T05:33:18.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: legacy client fscid: -1 2024-02-10T05:33:18.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:33:18.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: No filesystems configured 2024-02-10T05:33:18.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: mon.smithi028@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:33:18.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: mon.smithi028@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:33:18.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: mon.smithi028@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:33:18.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: mon.smithi028@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:33:18.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: mon.smithi028@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T05:33:18.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 bash[24662]: a5294cba5afaeca41b2bd8d72367849733d1306c18d2d8b9bde7ece6804f6ad8 2024-02-10T05:33:18.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 systemd[1]: Started Ceph mon.smithi028 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:33:18.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: mon.smithi028 is new leader, mons smithi028 in quorum (ranks 0) 2024-02-10T05:33:18.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: monmap e1: 1 mons at {smithi028=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0]} 2024-02-10T05:33:18.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: fsmap 2024-02-10T05:33:18.078 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T05:33:18.078 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:17 smithi028 ceph-mon[24680]: mgrmap e1: no daemons active 2024-02-10T05:33:19.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:19 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2475928692' entity='client.admin' 2024-02-10T05:33:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-10T05:33:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:33:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:Creating mgr... 2024-02-10T05:33:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:Verifying port 9283 ... 2024-02-10T05:33:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Failed to reset failed state of unit ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mgr.smithi028.ucuqfx.service: Unit ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mgr.smithi028.ucuqfx.service not loaded. 2024-02-10T05:33:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: Created symlink /etc/systemd/system/ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715.target.wants/ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mgr.smithi028.ucuqfx.service → /etc/systemd/system/ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@.service. 2024-02-10T05:33:27.780 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: disabled 2024-02-10T05:33:27.787 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: inactive 2024-02-10T05:33:27.788 INFO:teuthology.orchestra.run.smithi028.stderr:firewalld.service is not enabled 2024-02-10T05:33:27.788 INFO:teuthology.orchestra.run.smithi028.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T05:33:27.796 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: disabled 2024-02-10T05:33:27.802 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: inactive 2024-02-10T05:33:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:firewalld.service is not enabled 2024-02-10T05:33:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-10T05:33:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for mgr to start... 2024-02-10T05:33:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for mgr... 2024-02-10T05:33:28.335 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-02-10T05:33:28.335 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-02-10T05:33:28.335 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsid": "cf1ef3cc-c7d5-11ee-95b8-87774f69a715", 2024-02-10T05:33:28.335 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "health": { 2024-02-10T05:33:28.335 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T05:33:28.335 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 0 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "smithi028" 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-10T05:33:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T05:33:28.337 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:28.337 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T05:33:28.337 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:33:28.337 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T05:33:28.337 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T05:33:28.337 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T05:33:28.337 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T05:33:28.340 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T05:33:28.340 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T05:33:28.340 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:28.341 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T05:33:28.341 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T05:33:28.341 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T05:33:28.341 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T05:33:28.341 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T05:33:28.341 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T05:33:28.341 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T05:33:28.341 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T05:33:28.341 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T05:33:28.341 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "available": false, 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "iostat", 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "nfs", 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "restful" 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-02-10T05:33:28.342 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:28.343 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T05:33:28.343 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:33:28.343 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modified": "2024-02-10T05:33:13.237149+0000", 2024-02-10T05:33:28.343 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-02-10T05:33:28.343 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:28.343 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T05:33:28.343 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-02-10T05:33:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:mgr not available, waiting (1/15)... 2024-02-10T05:33:29.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:28 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2174655640' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:33:31.448 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-02-10T05:33:31.448 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-02-10T05:33:31.448 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsid": "cf1ef3cc-c7d5-11ee-95b8-87774f69a715", 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "health": { 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 0 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "smithi028" 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_age": 13, 2024-02-10T05:33:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T05:33:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:33:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-10T05:33:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T05:33:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T05:33:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:33:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T05:33:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T05:33:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T05:33:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T05:33:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "available": false, 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "iostat", 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "nfs", 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "restful" 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-02-10T05:33:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T05:33:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:33:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modified": "2024-02-10T05:33:13.237149+0000", 2024-02-10T05:33:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-02-10T05:33:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T05:33:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-02-10T05:33:31.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:31 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/851768194' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:33:32.083 INFO:teuthology.orchestra.run.smithi028.stderr:mgr not available, waiting (2/15)... 2024-02-10T05:33:34.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: Activating manager daemon smithi028.ucuqfx 2024-02-10T05:33:34.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: mgrmap e2: smithi028.ucuqfx(active, starting, since 0.00266829s) 2024-02-10T05:33:34.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: from='mgr.14100 172.21.15.28:0/1182030778' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:33:34.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: from='mgr.14100 172.21.15.28:0/1182030778' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:33:34.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: from='mgr.14100 172.21.15.28:0/1182030778' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:33:34.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: from='mgr.14100 172.21.15.28:0/1182030778' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:33:34.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: from='mgr.14100 172.21.15.28:0/1182030778' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi028.ucuqfx", "id": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:33:34.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: Manager daemon smithi028.ucuqfx is now available 2024-02-10T05:33:34.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: from='mgr.14100 172.21.15.28:0/1182030778' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:33:34.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: from='mgr.14100 172.21.15.28:0/1182030778' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:33:34.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: from='mgr.14100 172.21.15.28:0/1182030778' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:34.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: from='mgr.14100 172.21.15.28:0/1182030778' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:34.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:33 smithi028 ceph-mon[24680]: from='mgr.14100 172.21.15.28:0/1182030778' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:35.026 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-02-10T05:33:35.026 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsid": "cf1ef3cc-c7d5-11ee-95b8-87774f69a715", 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "health": { 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 0 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "smithi028" 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "quorum_age": 17, 2024-02-10T05:33:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T05:33:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:33:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-10T05:33:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T05:33:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T05:33:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:33:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T05:33:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T05:33:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T05:33:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T05:33:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "available": true, 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "iostat", 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "nfs", 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "restful" 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ], 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-02-10T05:33:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T05:33:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T05:33:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "modified": "2024-02-10T05:33:13.237149+0000", 2024-02-10T05:33:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "services": {} 2024-02-10T05:33:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: }, 2024-02-10T05:33:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T05:33:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-02-10T05:33:35.600 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:35 smithi028 ceph-mon[24680]: mgrmap e3: smithi028.ucuqfx(active, since 1.0071s) 2024-02-10T05:33:35.600 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:35 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2841672679' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:33:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:mgr is available 2024-02-10T05:33:36.200 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-02-10T05:33:36.200 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: [global] 2024-02-10T05:33:36.200 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: fsid = cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:33:36.200 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-10T05:33:36.200 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-10T05:33:36.200 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-10T05:33:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-10T05:33:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-02-10T05:33:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: [mgr] 2024-02-10T05:33:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-10T05:33:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 2024-02-10T05:33:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: [osd] 2024-02-10T05:33:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-10T05:33:36.201 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-10T05:33:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:Enabling cephadm module... 2024-02-10T05:33:36.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:36 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/45867333' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T05:33:37.784 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:37 smithi028 ceph-mon[24680]: mgrmap e4: smithi028.ucuqfx(active, since 3s) 2024-02-10T05:33:37.785 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:37 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2993778498' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T05:33:38.705 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:38 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2993778498' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T05:33:38.706 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:38 smithi028 ceph-mon[24680]: mgrmap e5: smithi028.ucuqfx(active, since 4s) 2024-02-10T05:33:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-02-10T05:33:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-10T05:33:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "available": true, 2024-02-10T05:33:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "active_name": "smithi028.ucuqfx", 2024-02-10T05:33:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T05:33:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-02-10T05:33:39.329 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for the mgr to restart... 2024-02-10T05:33:39.329 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for mgr epoch 5... 2024-02-10T05:33:39.565 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:39 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1133950969' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T05:33:44.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:43 smithi028 ceph-mon[24680]: Active manager daemon smithi028.ucuqfx restarted 2024-02-10T05:33:44.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:43 smithi028 ceph-mon[24680]: Activating manager daemon smithi028.ucuqfx 2024-02-10T05:33:44.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:43 smithi028 ceph-mon[24680]: osdmap e2: 0 total, 0 up, 0 in 2024-02-10T05:33:44.903 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-02-10T05:33:44.904 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-10T05:33:44.904 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T05:33:44.904 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-02-10T05:33:45.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: mgrmap e6: smithi028.ucuqfx(active, starting, since 0.440237s) 2024-02-10T05:33:45.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:33:45.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi028.ucuqfx", "id": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:33:45.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:33:45.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:33:45.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:33:45.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: Manager daemon smithi028.ucuqfx is now available 2024-02-10T05:33:45.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:45.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:45.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:33:45.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:33:45.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:33:45.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:33:45.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:44 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:33:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:mgr epoch 5 is available 2024-02-10T05:33:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:Setting orchestrator backend to cephadm... 2024-02-10T05:33:46.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:45 smithi028 ceph-mon[24680]: [10/Feb/2024:05:33:44] ENGINE Bus STARTING 2024-02-10T05:33:46.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:45 smithi028 ceph-mon[24680]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T05:33:46.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:45 smithi028 ceph-mon[24680]: mgrmap e7: smithi028.ucuqfx(active, since 1.44514s) 2024-02-10T05:33:46.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:45 smithi028 ceph-mon[24680]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T05:33:46.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:45 smithi028 ceph-mon[24680]: [10/Feb/2024:05:33:44] ENGINE Serving on https://172.21.15.28:7150 2024-02-10T05:33:46.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:45 smithi028 ceph-mon[24680]: [10/Feb/2024:05:33:44] ENGINE Bus STARTED 2024-02-10T05:33:46.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:45 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:46.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:45 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:33:47.162 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:46 smithi028 ceph-mon[24680]: mgrmap e8: smithi028.ucuqfx(active, since 2s) 2024-02-10T05:33:47.162 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:46 smithi028 ceph-mon[24680]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:33:47.162 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:46 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:47.162 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:46 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:33:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: value unchanged 2024-02-10T05:33:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:Generating ssh key... 2024-02-10T05:33:48.002 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:47 smithi028 ceph-mon[24680]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:33:49.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:48 smithi028 ceph-mon[24680]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:33:49.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:48 smithi028 ceph-mon[24680]: Generating ssh key... 2024-02-10T05:33:49.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:48 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:49.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:48 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQClzWRJY1gl2YwG3YsaaWoUDQlH3IIhlqFXl4gZIAf5x3qCfbVqGUiO8w1qpKJYNrayepqWn0x3Eh0GVxVNQWlfAO5/1MP4SiDLk774T9qGZ3Rur/ogsUTYySbgA3JxgraqfDgQU/Up/67KwCNDQK/CwCOaJGxP7MYo8dRNqyuXLrwnqbKcRGJwZ+kI1I7ATu/k9DRqJ+9dyb0R33HtDvaYsTVVU4hyh1NKevU3TYRfEqn2bYE6Nb4ToIrvgKERgyzYFOGxCtbK/ICycuQ559Bc5bUIL3Inycihxcd+U+GWYtLoI7+4hI9weo4pLF7ulERYBE04vb7JPzjW4pNAepYcM+fYtaml68eBeg2bpD4F1toJ2CUWvLjHJv7wIhgzVxsHvwwwOYrnPpCyarRhsNfDIpgMIhY5efAtTjwFR7fq+SYlJxsLDbNKLUenS0LxaKsZBnRZenSznsd/GO+pC3stryP47qHUyYbb0lGCmoDQmDj2SsxjEkasefVmHXJnJBU= ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:33:50.475 INFO:teuthology.orchestra.run.smithi028.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T05:33:50.476 INFO:teuthology.orchestra.run.smithi028.stderr:Adding key to root@localhost authorized_keys... 2024-02-10T05:33:50.476 INFO:teuthology.orchestra.run.smithi028.stderr:Adding host smithi028... 2024-02-10T05:33:50.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:50 smithi028 ceph-mon[24680]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:33:52.045 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:51 smithi028 ceph-mon[24680]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:33:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Added host 'smithi028' with addr '172.21.15.28' 2024-02-10T05:33:52.863 INFO:teuthology.orchestra.run.smithi028.stderr:Deploying mon service with default placement... 2024-02-10T05:33:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-10T05:33:53.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:53 smithi028 ceph-mon[24680]: Deploying cephadm binary to smithi028 2024-02-10T05:33:53.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:53 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:53.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:53 smithi028 ceph-mon[24680]: Added host smithi028 2024-02-10T05:33:53.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:53 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:33:54.138 INFO:teuthology.orchestra.run.smithi028.stderr:Deploying mgr service with default placement... 2024-02-10T05:33:54.724 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:54 smithi028 ceph-mon[24680]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:33:54.724 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:54 smithi028 ceph-mon[24680]: Saving service mon spec with placement count:5 2024-02-10T05:33:54.724 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:54 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-10T05:33:56.044 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:55 smithi028 ceph-mon[24680]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:33:56.044 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:55 smithi028 ceph-mon[24680]: Saving service mgr spec with placement count:2 2024-02-10T05:33:56.044 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:55 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:56.044 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:55 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:56.116 INFO:teuthology.orchestra.run.smithi028.stderr:Deploying crash service with default placement... 2024-02-10T05:33:56.747 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-10T05:33:57.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:57 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:57.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:57 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:Deploying prometheus service with default placement... 2024-02-10T05:33:57.962 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-10T05:33:58.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:58 smithi028 ceph-mon[24680]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:33:58.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:58 smithi028 ceph-mon[24680]: Saving service crash spec with placement * 2024-02-10T05:33:58.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:58 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:Deploying grafana service with default placement... 2024-02-10T05:33:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-10T05:33:59.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:59 smithi028 ceph-mon[24680]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:33:59.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:59 smithi028 ceph-mon[24680]: Saving service prometheus spec with placement count:1 2024-02-10T05:33:59.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:59 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:59.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:33:59 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:33:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:Deploying node-exporter service with default placement... 2024-02-10T05:34:00.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:00 smithi028 ceph-mon[24680]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:34:00.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:00 smithi028 ceph-mon[24680]: Saving service grafana spec with placement count:1 2024-02-10T05:34:00.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:00 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:01.257 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-10T05:34:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:Deploying alertmanager service with default placement... 2024-02-10T05:34:02.426 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:02 smithi028 ceph-mon[24680]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:34:02.426 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:02 smithi028 ceph-mon[24680]: Saving service node-exporter spec with placement * 2024-02-10T05:34:02.426 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:02 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-10T05:34:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:03 smithi028 ceph-mon[24680]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:34:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:03 smithi028 ceph-mon[24680]: Saving service alertmanager spec with placement count:1 2024-02-10T05:34:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:03 smithi028 ceph-mon[24680]: from='mgr.14118 172.21.15.28:0/439401803' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:05.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:05 smithi028 ceph-mon[24680]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:05.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:05 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/94543105' entity='client.admin' 2024-02-10T05:34:05.925 INFO:teuthology.orchestra.run.smithi028.stderr:Enabling the dashboard module... 2024-02-10T05:34:06.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:06 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/471162469' entity='client.admin' 2024-02-10T05:34:07.678 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:07 smithi028 ceph-mon[24680]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:07.678 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:07 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3323483709' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T05:34:08.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:08 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3323483709' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T05:34:08.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:08 smithi028 ceph-mon[24680]: mgrmap e9: smithi028.ucuqfx(active, since 23s) 2024-02-10T05:34:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-02-10T05:34:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-10T05:34:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "available": true, 2024-02-10T05:34:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "active_name": "smithi028.ucuqfx", 2024-02-10T05:34:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T05:34:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-02-10T05:34:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for the mgr to restart... 2024-02-10T05:34:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:Waiting for mgr epoch 9... 2024-02-10T05:34:09.464 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:09 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2584599395' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T05:34:14.759 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: Active manager daemon smithi028.ucuqfx restarted 2024-02-10T05:34:14.759 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: Activating manager daemon smithi028.ucuqfx 2024-02-10T05:34:14.759 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: osdmap e3: 0 total, 0 up, 0 in 2024-02-10T05:34:14.759 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: mgrmap e10: smithi028.ucuqfx(active, starting, since 0.521528s) 2024-02-10T05:34:14.759 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:34:14.759 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi028.ucuqfx", "id": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:34:14.759 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:34:14.759 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:34:14.759 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:34:14.760 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: Manager daemon smithi028.ucuqfx is now available 2024-02-10T05:34:14.760 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:34:14.760 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:34:14.760 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:34:14.760 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:14 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:34:14.909 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: { 2024-02-10T05:34:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-10T05:34:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T05:34:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: } 2024-02-10T05:34:15.697 INFO:teuthology.orchestra.run.smithi028.stderr:mgr epoch 9 is available 2024-02-10T05:34:15.697 INFO:teuthology.orchestra.run.smithi028.stderr:Generating a dashboard self-signed certificate... 2024-02-10T05:34:15.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:15 smithi028 ceph-mon[24680]: mgrmap e11: smithi028.ucuqfx(active, since 1.52455s) 2024-02-10T05:34:15.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:15 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:15.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:15 smithi028 ceph-mon[24680]: [10/Feb/2024:05:34:15] ENGINE Bus STARTING 2024-02-10T05:34:15.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:15 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-10T05:34:17.062 INFO:teuthology.orchestra.run.smithi028.stderr:Creating initial admin user... 2024-02-10T05:34:17.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:17 smithi028 ceph-mon[24680]: [10/Feb/2024:05:34:15] ENGINE Serving on https://172.21.15.28:7150 2024-02-10T05:34:17.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:17 smithi028 ceph-mon[24680]: [10/Feb/2024:05:34:15] ENGINE Bus STARTED 2024-02-10T05:34:17.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:17 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:17.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:17 smithi028 ceph-mon[24680]: mgrmap e12: smithi028.ucuqfx(active, since 2s) 2024-02-10T05:34:17.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:17 smithi028 ceph-mon[24680]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:34:17.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:17 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:17.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:17 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:17.960 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$/TpPxy7AAvZTMbJJ.Uv1yOkA4LOpANCJOtg62d4RlgbZEBSJ9izQ6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707543257, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T05:34:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:Fetching dashboard port number... 2024-02-10T05:34:19.226 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:18 smithi028 ceph-mon[24680]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:34:19.227 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:18 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:19.742 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: 8443 2024-02-10T05:34:20.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:19 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2781544526' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T05:34:20.341 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: disabled 2024-02-10T05:34:20.348 INFO:teuthology.orchestra.run.smithi028.stderr:systemctl: inactive 2024-02-10T05:34:20.349 INFO:teuthology.orchestra.run.smithi028.stderr:firewalld.service is not enabled 2024-02-10T05:34:20.349 INFO:teuthology.orchestra.run.smithi028.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T05:34:20.351 INFO:teuthology.orchestra.run.smithi028.stderr:Ceph Dashboard is now available at: 2024-02-10T05:34:20.352 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-10T05:34:20.352 INFO:teuthology.orchestra.run.smithi028.stderr: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-02-10T05:34:20.352 INFO:teuthology.orchestra.run.smithi028.stderr: User: admin 2024-02-10T05:34:20.352 INFO:teuthology.orchestra.run.smithi028.stderr: Password: 69j0vcp743 2024-02-10T05:34:20.352 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-10T05:34:20.352 INFO:teuthology.orchestra.run.smithi028.stderr:Enabling autotune for osd_memory_target 2024-02-10T05:34:21.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:20 smithi028 ceph-mon[24680]: mgrmap e13: smithi028.ucuqfx(active, since 6s) 2024-02-10T05:34:22.472 INFO:teuthology.orchestra.run.smithi028.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-10T05:34:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:You can access the Ceph CLI with: 2024-02-10T05:34:23.018 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-10T05:34:23.018 INFO:teuthology.orchestra.run.smithi028.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:34:23.018 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-10T05:34:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-10T05:34:23.018 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-10T05:34:23.018 INFO:teuthology.orchestra.run.smithi028.stderr: ceph telemetry on 2024-02-10T05:34:23.018 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-10T05:34:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:For more information see: 2024-02-10T05:34:23.018 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-10T05:34:23.019 INFO:teuthology.orchestra.run.smithi028.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-10T05:34:23.019 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-10T05:34:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:Bootstrap complete. 2024-02-10T05:34:23.054 INFO:tasks.cephadm:Fetching config... 2024-02-10T05:34:23.054 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-10T05:34:23.054 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T05:34:23.070 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T05:34:23.070 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-10T05:34:23.070 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T05:34:23.125 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T05:34:23.125 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-10T05:34:23.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/keyring of=/dev/stdout 2024-02-10T05:34:23.192 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T05:34:23.192 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-10T05:34:23.192 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T05:34:23.247 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T05:34:23.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQClzWRJY1gl2YwG3YsaaWoUDQlH3IIhlqFXl4gZIAf5x3qCfbVqGUiO8w1qpKJYNrayepqWn0x3Eh0GVxVNQWlfAO5/1MP4SiDLk774T9qGZ3Rur/ogsUTYySbgA3JxgraqfDgQU/Up/67KwCNDQK/CwCOaJGxP7MYo8dRNqyuXLrwnqbKcRGJwZ+kI1I7ATu/k9DRqJ+9dyb0R33HtDvaYsTVVU4hyh1NKevU3TYRfEqn2bYE6Nb4ToIrvgKERgyzYFOGxCtbK/ICycuQ559Bc5bUIL3Inycihxcd+U+GWYtLoI7+4hI9weo4pLF7ulERYBE04vb7JPzjW4pNAepYcM+fYtaml68eBeg2bpD4F1toJ2CUWvLjHJv7wIhgzVxsHvwwwOYrnPpCyarRhsNfDIpgMIhY5efAtTjwFR7fq+SYlJxsLDbNKLUenS0LxaKsZBnRZenSznsd/GO+pC3stryP47qHUyYbb0lGCmoDQmDj2SsxjEkasefVmHXJnJBU= ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T05:34:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQClzWRJY1gl2YwG3YsaaWoUDQlH3IIhlqFXl4gZIAf5x3qCfbVqGUiO8w1qpKJYNrayepqWn0x3Eh0GVxVNQWlfAO5/1MP4SiDLk774T9qGZ3Rur/ogsUTYySbgA3JxgraqfDgQU/Up/67KwCNDQK/CwCOaJGxP7MYo8dRNqyuXLrwnqbKcRGJwZ+kI1I7ATu/k9DRqJ+9dyb0R33HtDvaYsTVVU4hyh1NKevU3TYRfEqn2bYE6Nb4ToIrvgKERgyzYFOGxCtbK/ICycuQ559Bc5bUIL3Inycihxcd+U+GWYtLoI7+4hI9weo4pLF7ulERYBE04vb7JPzjW4pNAepYcM+fYtaml68eBeg2bpD4F1toJ2CUWvLjHJv7wIhgzVxsHvwwwOYrnPpCyarRhsNfDIpgMIhY5efAtTjwFR7fq+SYlJxsLDbNKLUenS0LxaKsZBnRZenSznsd/GO+pC3stryP47qHUyYbb0lGCmoDQmDj2SsxjEkasefVmHXJnJBU= ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:34:23.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQClzWRJY1gl2YwG3YsaaWoUDQlH3IIhlqFXl4gZIAf5x3qCfbVqGUiO8w1qpKJYNrayepqWn0x3Eh0GVxVNQWlfAO5/1MP4SiDLk774T9qGZ3Rur/ogsUTYySbgA3JxgraqfDgQU/Up/67KwCNDQK/CwCOaJGxP7MYo8dRNqyuXLrwnqbKcRGJwZ+kI1I7ATu/k9DRqJ+9dyb0R33HtDvaYsTVVU4hyh1NKevU3TYRfEqn2bYE6Nb4ToIrvgKERgyzYFOGxCtbK/ICycuQ559Bc5bUIL3Inycihxcd+U+GWYtLoI7+4hI9weo4pLF7ulERYBE04vb7JPzjW4pNAepYcM+fYtaml68eBeg2bpD4F1toJ2CUWvLjHJv7wIhgzVxsHvwwwOYrnPpCyarRhsNfDIpgMIhY5efAtTjwFR7fq+SYlJxsLDbNKLUenS0LxaKsZBnRZenSznsd/GO+pC3stryP47qHUyYbb0lGCmoDQmDj2SsxjEkasefVmHXJnJBU= ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T05:34:23.407 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQClzWRJY1gl2YwG3YsaaWoUDQlH3IIhlqFXl4gZIAf5x3qCfbVqGUiO8w1qpKJYNrayepqWn0x3Eh0GVxVNQWlfAO5/1MP4SiDLk774T9qGZ3Rur/ogsUTYySbgA3JxgraqfDgQU/Up/67KwCNDQK/CwCOaJGxP7MYo8dRNqyuXLrwnqbKcRGJwZ+kI1I7ATu/k9DRqJ+9dyb0R33HtDvaYsTVVU4hyh1NKevU3TYRfEqn2bYE6Nb4ToIrvgKERgyzYFOGxCtbK/ICycuQ559Bc5bUIL3Inycihxcd+U+GWYtLoI7+4hI9weo4pLF7ulERYBE04vb7JPzjW4pNAepYcM+fYtaml68eBeg2bpD4F1toJ2CUWvLjHJv7wIhgzVxsHvwwwOYrnPpCyarRhsNfDIpgMIhY5efAtTjwFR7fq+SYlJxsLDbNKLUenS0LxaKsZBnRZenSznsd/GO+pC3stryP47qHUyYbb0lGCmoDQmDj2SsxjEkasefVmHXJnJBU= ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:34:23.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T05:34:23.731 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:23 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1630003879' entity='client.admin' 2024-02-10T05:34:26.093 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:25 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:26.093 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:25 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:34:26.093 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:25 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:26.093 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:25 smithi028 ceph-mon[24680]: Deploying daemon alertmanager.smithi028 on smithi028 2024-02-10T05:34:26.767 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-10T05:34:26.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-10T05:34:27.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:27 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1460451540' entity='client.admin' 2024-02-10T05:34:30.259 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2024-02-10T05:34:30.259 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-10T05:34:30.259 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2024-02-10T05:34:30.279 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-10T05:34:30.279 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:34:30.338 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2024-02-10T05:34:30.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch host add smithi192 2024-02-10T05:34:30.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:30 smithi028 ceph-mon[24680]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:34:30.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:30 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:35.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:34 smithi028 ceph-mon[24680]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:35.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:34 smithi028 ceph-mon[24680]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:34:35.299 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi192' with addr '172.21.15.192' 2024-02-10T05:34:35.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T05:34:36.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:36 smithi028 ceph-mon[24680]: Deploying cephadm binary to smithi192 2024-02-10T05:34:36.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:36 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:36.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:36 smithi028 ceph-mon[24680]: Added host smithi192 2024-02-10T05:34:37.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:37 smithi028 ceph-mon[24680]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:38.241 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:34:38.241 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2024-02-10T05:34:39.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:38 smithi028 ceph-mon[24680]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:39.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:38 smithi028 ceph-mon[24680]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:34:39.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:38 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:39.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:38 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi028", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:34:39.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:38 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi028", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-10T05:34:39.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:38 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:34:39.383 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T05:34:39.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T05:34:40.171 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:39 smithi028 ceph-mon[24680]: Deploying daemon crash.smithi028 on smithi028 2024-02-10T05:34:40.171 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:39 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:41.125 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:40 smithi028 ceph-mon[24680]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:42.959 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-02-10T05:34:43.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:42 smithi028 ceph-mon[24680]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:43.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:42 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/99866185' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T05:34:43.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:42 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:43.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:42 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:43.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:42 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:43.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:42 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:34:43.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:42 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:43.572 INFO:tasks.cephadm:Adding mon.smithi028 on smithi028 2024-02-10T05:34:43.572 INFO:tasks.cephadm:Adding mon.smithi192 on smithi192 2024-02-10T05:34:43.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch apply mon '2;smithi028:172.21.15.28=smithi028;smithi192:172.21.15.192=smithi192' 2024-02-10T05:34:44.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:43 smithi028 ceph-mon[24680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:34:44.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:43 smithi028 ceph-mon[24680]: Deploying daemon grafana.smithi028 on smithi028 2024-02-10T05:34:44.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:43 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/99866185' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T05:34:44.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:43 smithi028 ceph-mon[24680]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:34:44.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:43 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:45.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:44 smithi028 ceph-mon[24680]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:45.475 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2024-02-10T05:34:46.149 DEBUG:teuthology.orchestra.run.smithi192:mon.smithi192> sudo journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi192.service 2024-02-10T05:34:46.152 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T05:34:46.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:34:46.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:46 smithi028 ceph-mon[24680]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi028:172.21.15.28=smithi028;smithi192:172.21.15.192=smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:34:46.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:46 smithi028 ceph-mon[24680]: Saving service mon spec with placement smithi028:172.21.15.28=smithi028;smithi192:172.21.15.192=smithi192;count:2 2024-02-10T05:34:46.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:46 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:46.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:46 smithi028 ceph-mon[24680]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:48.000 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:34:48.000 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","modified":"2024-02-10T05:33:10.477688Z","created":"2024-02-10T05:33:10.477688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi028","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:34:48.002 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-10T05:34:49.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:48 smithi028 ceph-mon[24680]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:48 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/4283678155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:34:49.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T05:34:49.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:34:51.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:50 smithi028 ceph-mon[24680]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:51.606 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:34:51.607 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","modified":"2024-02-10T05:33:10.477688Z","created":"2024-02-10T05:33:10.477688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi028","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:34:51.608 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-10T05:34:52.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:51 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/3509761791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:34:53.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T05:34:53.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:34:53.244 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:52 smithi028 ceph-mon[24680]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:55.180 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:34:55.180 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","modified":"2024-02-10T05:33:10.477688Z","created":"2024-02-10T05:33:10.477688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi028","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:34:55.181 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-10T05:34:55.303 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:54 smithi028 ceph-mon[24680]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:56.200 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:55 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/2785244384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:34:56.200 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:55 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:56.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T05:34:56.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:34:57.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:56 smithi028 ceph-mon[24680]: Deploying daemon node-exporter.smithi028 on smithi028 2024-02-10T05:34:57.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:56 smithi028 ceph-mon[24680]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:58.670 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:34:58.670 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","modified":"2024-02-10T05:33:10.477688Z","created":"2024-02-10T05:33:10.477688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi028","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:34:58.672 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-10T05:34:59.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:58 smithi028 ceph-mon[24680]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:34:59.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:58 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/2637115834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:34:59.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:58 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:34:59.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:58 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:35:00.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:59 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:00.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:59 smithi028 ceph-mon[24680]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:35:00.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:34:59 smithi028 ceph-mon[24680]: mgrmap e14: smithi028.ucuqfx(active, since 46s) 2024-02-10T05:35:00.270 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T05:35:00.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:35:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:35:02.236 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","modified":"2024-02-10T05:33:10.477688Z","created":"2024-02-10T05:33:10.477688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi028","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:35:02.238 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-10T05:35:02.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:02 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/3520995225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:03.831 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T05:35:03.831 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:35:05.727 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:35:05.727 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","modified":"2024-02-10T05:33:10.477688Z","created":"2024-02-10T05:33:10.477688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi028","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:35:05.728 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-10T05:35:05.792 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:05 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/348816014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:07.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:06 smithi028 ceph-mon[24680]: Active manager daemon smithi028.ucuqfx restarted 2024-02-10T05:35:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:06 smithi028 ceph-mon[24680]: Activating manager daemon smithi028.ucuqfx 2024-02-10T05:35:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:06 smithi028 ceph-mon[24680]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T05:35:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:06 smithi028 ceph-mon[24680]: mgrmap e15: smithi028.ucuqfx(active, starting, since 0.974365s) 2024-02-10T05:35:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:35:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi028.ucuqfx", "id": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:35:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:35:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:35:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:35:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:06 smithi028 ceph-mon[24680]: Manager daemon smithi028.ucuqfx is now available 2024-02-10T05:35:07.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T05:35:07.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:35:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:35:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:35:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:35:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:35:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:35:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:07 smithi028 ceph-mon[24680]: mgrmap e16: smithi028.ucuqfx(active, since 1.9763s) 2024-02-10T05:35:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:09.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:08 smithi028 ceph-mon[24680]: [10/Feb/2024:05:35:07] ENGINE Bus STARTING 2024-02-10T05:35:09.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:08 smithi028 ceph-mon[24680]: [10/Feb/2024:05:35:07] ENGINE Serving on https://172.21.15.28:7150 2024-02-10T05:35:09.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:08 smithi028 ceph-mon[24680]: [10/Feb/2024:05:35:07] ENGINE Bus STARTED 2024-02-10T05:35:09.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:08 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:09.344 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:35:09.344 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","modified":"2024-02-10T05:33:10.477688Z","created":"2024-02-10T05:33:10.477688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi028","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:35:09.347 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-10T05:35:10.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:09 smithi028 ceph-mon[24680]: mgrmap e17: smithi028.ucuqfx(active, since 3s) 2024-02-10T05:35:10.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:09 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:10.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:09 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/3980278896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:10.932 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T05:35:10.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:35:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:13 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:13 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:13 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:13 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:35:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:13 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:13 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:35:13.613 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:35:13.613 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","modified":"2024-02-10T05:33:10.477688Z","created":"2024-02-10T05:33:10.477688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi028","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:35:13.615 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-10T05:35:14.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:14 smithi028 ceph-mon[24680]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-10T05:35:14.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:14 smithi028 ceph-mon[24680]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:35:14.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:14.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:14.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:35:14.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-10T05:35:14.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:14.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:14 smithi028 ceph-mon[24680]: Deploying daemon crash.smithi192 on smithi192 2024-02-10T05:35:14.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:14 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/3178796175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:15.213 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T05:35:15.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:35:17.405 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:35:17.406 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","modified":"2024-02-10T05:33:10.477688Z","created":"2024-02-10T05:33:10.477688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi028","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:35:17.407 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-10T05:35:17.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:17 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/3423177857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:19.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:19.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.hppzcm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:35:19.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.hppzcm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:35:19.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:35:19.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:19.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:18 smithi028 ceph-mon[24680]: Deploying daemon mgr.smithi192.hppzcm on smithi192 2024-02-10T05:35:19.069 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T05:35:19.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:35:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:35:20.973 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","modified":"2024-02-10T05:33:10.477688Z","created":"2024-02-10T05:33:10.477688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi028","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:35:20.976 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-02-10T05:35:21.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:21 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/2840504348' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:22.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:22 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:22.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:22 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:35:22.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:22 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:22.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:22 smithi028 ceph-mon[24680]: Deploying daemon mon.smithi192 on smithi192 2024-02-10T05:35:22.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:22 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:22.733 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T05:35:22.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:35:25.397 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 podman[26879]: 2024-02-10 05:35:25.277142333 +0000 UTC m=+0.019400490 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-02-10T05:35:25.397 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 podman[26879]: 2024-02-10 05:35:25.396099627 +0000 UTC m=+0.138357770 container create cf2f3de335d9857551c02971eba2548a906005a8b6efe478a3ce553e8ec9414d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192, release=754, vendor=Red Hat, Inc., GIT_CLEAN=True, architecture=x86_64, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git) 2024-02-10T05:35:25.646 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 podman[26879]: 2024-02-10 05:35:25.615411663 +0000 UTC m=+0.357669807 container init cf2f3de335d9857551c02971eba2548a906005a8b6efe478a3ce553e8ec9414d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192, distribution-scope=public, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., architecture=x86_64, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-02-10T05:35:25.647 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 podman[26879]: 2024-02-10 05:35:25.618407708 +0000 UTC m=+0.360665852 container start cf2f3de335d9857551c02971eba2548a906005a8b6efe478a3ce553e8ec9414d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, vendor=Red Hat, Inc., RELEASE=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.expose-services=, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-10T05:35:25.647 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:35:25.647 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-10T05:35:25.647 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: pidfile_write: ignore empty --pid-file 2024-02-10T05:35:25.647 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: load: jerasure load: lrc 2024-02-10T05:35:25.647 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: RocksDB version: 6.15.5 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Compile date Apr 18 2022 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: DB SUMMARY 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: DB Session ID: JHERGP64HUHCMHU46O4U 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi192/store.db dir, Total Num: 0, files: 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi192/store.db: 000004.log size: 511 ; 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:35:25.648 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.env: 0x56419f4c6860 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.fs: Posix File System 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.info_log: 0x5641a1247fa0 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.statistics: (nil) 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.use_fsync: 0 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:35:25.649 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.db_log_dir: 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi192/store.db 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.write_buffer_manager: 0x5641a1338330 2024-02-10T05:35:25.650 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.unordered_write: 0 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:35:25.651 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.row_cache: None 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.wal_filter: None 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.preserve_deletes: 0 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:35:25.652 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:35:25.653 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:35:25.653 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:35:25.653 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:35:25.653 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:35:25.653 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:35:25.653 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:35:25.653 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:35:25.653 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:35:25.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:35:25.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:35:25.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:35:25.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:35:25.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:35:25.654 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_open_files: -1 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Compression algorithms supported: 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: kZSTD supported: 0 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:35:25.655 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi192/store.db/MANIFEST-000003 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.merge_operator: 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_filter: None 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5641a1213d00) 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:35:25.656 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: index_type: 0 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: data_block_index_type: 0 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: index_shortening: 1 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: hash_index_allow_collision: 1 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: checksum: 1 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: no_block_cache: 0 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_cache: 0x5641a127e170 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_cache_options: 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: capacity : 536870912 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: num_shard_bits : 4 2024-02-10T05:35:25.657 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: strict_capacity_limit : 0 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_cache_compressed: (nil) 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: persistent_cache: (nil) 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_size: 4096 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_size_deviation: 10 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_restart_interval: 16 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: index_block_restart_interval: 1 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: metadata_block_size: 4096 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: partition_filters: 0 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: use_delta_encoding: 1 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: whole_key_filtering: 1 2024-02-10T05:35:25.658 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: verify_compression: 0 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: format_version: 4 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: enable_index_compression: 1 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_align: 0 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compression: NoCompression 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.num_levels: 7 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:35:25.659 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:35:25.660 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:35:25.661 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.arena_block_size: 4194304 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:35:25.662 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:35:25.663 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.ttl: 2592000 2024-02-10T05:35:25.664 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:35:25.664 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:35:25.664 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:35:25.664 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:35:25.664 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:35:25.664 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:35:25.664 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:35:25.664 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi192/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T05:35:25.664 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-02-10T05:35:25.664 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-02-10T05:35:25.664 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543325644832, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543325645154, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707543325, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "493ca81f-8e72-4ea0-b413-27bd089750fa", "db_session_id": "JHERGP64HUHCMHU46O4U"}} 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543325645461, "job": 1, "event": "recovery_finished"} 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi192/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5641a1264a80 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: DB pointer 0x5641a1274000 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** DB Stats ** 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:35:25.665 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** Compaction Stats [default] ** 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** Compaction Stats [default] ** 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:35:25.666 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative compaction: 0.00 GB write, 0.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval compaction: 0.00 GB write, 0.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** Compaction Stats [default] ** 2024-02-10T05:35:25.667 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** Compaction Stats [default] ** 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:35:25.668 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:35:25.669 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:35:25.669 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:35:25.669 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:35:25.669 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:35:25.669 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:35:25.669 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:35:25.669 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192 does not exist in monmap, will attempt to join an existing cluster 2024-02-10T05:35:25.669 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: using public_addr v2:172.21.15.192:0/0 -> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2024-02-10T05:35:25.669 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: starting mon.smithi192 rank -1 at public addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] at bind addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi192 fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:35:25.669 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(???) e0 preinit fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:35:25.901 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).mds e1 new map 2024-02-10T05:35:25.901 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).mds e1 print_map 2024-02-10T05:35:25.901 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: e1 2024-02-10T05:35:25.901 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:35:25.901 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T05:35:25.901 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: legacy client fscid: -1 2024-02-10T05:35:25.901 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:35:25.901 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: No filesystems configured 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: Deploying daemon crash.smithi028 on smithi028 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.902 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/99866185' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: Deploying daemon grafana.smithi028 on smithi028 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/99866185' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi028:172.21.15.28=smithi028;smithi192:172.21.15.192=smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: Saving service mon spec with placement smithi028:172.21.15.28=smithi028;smithi192:172.21.15.192=smithi192;count:2 2024-02-10T05:35:25.903 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/4283678155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/3509761791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/2785244384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: Deploying daemon node-exporter.smithi028 on smithi028 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/2637115834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:35:25.905 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.905 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14160 172.21.15.28:0/2524816934' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:35:25.905 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mgrmap e14: smithi028.ucuqfx(active, since 46s) 2024-02-10T05:35:25.905 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/3520995225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:25.905 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/348816014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:25.905 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: Active manager daemon smithi028.ucuqfx restarted 2024-02-10T05:35:25.905 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: Activating manager daemon smithi028.ucuqfx 2024-02-10T05:35:25.905 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T05:35:25.905 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mgrmap e15: smithi028.ucuqfx(active, starting, since 0.974365s) 2024-02-10T05:35:25.905 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:35:25.905 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi028.ucuqfx", "id": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:35:25.906 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:35:25.906 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:35:25.906 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:35:25.906 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: Manager daemon smithi028.ucuqfx is now available 2024-02-10T05:35:25.906 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:35:25.906 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:35:25.906 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:25.906 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:35:25.906 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.906 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:35:25.906 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:35:25.906 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mgrmap e16: smithi028.ucuqfx(active, since 1.9763s) 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: [10/Feb/2024:05:35:07] ENGINE Bus STARTING 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: [10/Feb/2024:05:35:07] ENGINE Serving on https://172.21.15.28:7150 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: [10/Feb/2024:05:35:07] ENGINE Bus STARTED 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mgrmap e17: smithi028.ucuqfx(active, since 3s) 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/3980278896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.907 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: Deploying daemon crash.smithi192 on smithi192 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/3178796175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/3423177857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.hppzcm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.hppzcm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:35:25.908 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:25.909 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: Deploying daemon mgr.smithi192.hppzcm on smithi192 2024-02-10T05:35:25.909 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/2840504348' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:25.909 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.909 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:35:25.909 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:25.909 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: Deploying daemon mon.smithi192 on smithi192 2024-02-10T05:35:25.909 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:25.909 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-02-10T05:35:25.909 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 ceph-mon[26904]: mon.smithi192@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-10T05:35:25.909 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 bash[26879]: cf2f3de335d9857551c02971eba2548a906005a8b6efe478a3ce553e8ec9414d 2024-02-10T05:35:25.909 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:25 smithi192 systemd[1]: Started Ceph mon.smithi192 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:35:31.023 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:35:31.023 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: mon.smithi028 calling monitor election 2024-02-10T05:35:31.023 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:31.023 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:31.023 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:31.023 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/1152729814' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/crt"}]: dispatch 2024-02-10T05:35:31.023 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:31.023 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: mon.smithi192 calling monitor election 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: mon.smithi028 is new leader, mons smithi028,smithi192 in quorum (ranks 0,1) 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: monmap e2: 2 mons at {smithi028=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: fsmap 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: mgrmap e17: smithi028.ucuqfx(active, since 24s) 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: Standby manager daemon smithi192.hppzcm started 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/1152729814' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: overall HEALTH_OK 2024-02-10T05:35:31.024 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/1152729814' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/key"}]: dispatch 2024-02-10T05:35:31.025 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:31.025 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:30 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/1152729814' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:35:31.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:35:31.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: mon.smithi028 calling monitor election 2024-02-10T05:35:31.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:31.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:31.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:31.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/1152729814' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/crt"}]: dispatch 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: mon.smithi192 calling monitor election 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: mon.smithi028 is new leader, mons smithi028,smithi192 in quorum (ranks 0,1) 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: monmap e2: 2 mons at {smithi028=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: fsmap 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: osdmap e5: 0 total, 0 up, 0 in 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: mgrmap e17: smithi028.ucuqfx(active, since 24s) 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: Standby manager daemon smithi192.hppzcm started 2024-02-10T05:35:31.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/1152729814' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:35:31.061 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: overall HEALTH_OK 2024-02-10T05:35:31.061 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/1152729814' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/key"}]: dispatch 2024-02-10T05:35:31.061 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:31.061 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:30 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/1152729814' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:35:32.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:31 smithi028 ceph-mon[24680]: Deploying daemon node-exporter.smithi192 on smithi192 2024-02-10T05:35:32.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:31 smithi028 ceph-mon[24680]: mgrmap e18: smithi028.ucuqfx(active, since 24s), standbys: smithi192.hppzcm 2024-02-10T05:35:32.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:31 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi192.hppzcm", "id": "smithi192.hppzcm"}]: dispatch 2024-02-10T05:35:32.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:31 smithi028 ceph-mon[24680]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:32.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:31 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:32.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:31 smithi192 ceph-mon[26904]: Deploying daemon node-exporter.smithi192 on smithi192 2024-02-10T05:35:32.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:31 smithi192 ceph-mon[26904]: mgrmap e18: smithi028.ucuqfx(active, since 24s), standbys: smithi192.hppzcm 2024-02-10T05:35:32.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:31 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi192.hppzcm", "id": "smithi192.hppzcm"}]: dispatch 2024-02-10T05:35:32.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:31 smithi192 ceph-mon[26904]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:32.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:31 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:35:32.273 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-02-10T05:35:32.273 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","modified":"2024-02-10T05:35:25.670620Z","created":"2024-02-10T05:33:10.477688Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi028","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi192","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T05:35:32.274 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2024-02-10T05:35:33.209 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:32 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:33.210 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:32 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/3554478248' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:33.210 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:32 smithi028 ceph-mon[24680]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:32 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:32 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/3554478248' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:35:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:32 smithi192 ceph-mon[26904]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:33.428 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-10T05:35:33.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-10T05:35:35.313 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:35:35.314 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-02-10T05:35:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:35:35.314 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2024-02-10T05:35:35.754 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:35 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:35.754 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:35 smithi028 ceph-mon[24680]: Deploying daemon prometheus.smithi028 on smithi028 2024-02-10T05:35:35.754 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:35 smithi028 ceph-mon[24680]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:35.754 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:35 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2151994509' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:35.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:35 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:35.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:35 smithi192 ceph-mon[26904]: Deploying daemon prometheus.smithi028 on smithi028 2024-02-10T05:35:35.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:35 smithi192 ceph-mon[26904]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:35.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:35 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2151994509' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:35.845 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-10T05:35:35.845 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-10T05:35:35.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T05:35:35.876 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-10T05:35:35.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:35:35.944 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-10T05:35:35.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T05:35:35.981 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-10T05:35:35.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:35:36.052 INFO:tasks.cephadm:Deploying OSDs... 2024-02-10T05:35:36.052 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-10T05:35:36.052 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:35:36.067 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T05:35:36.068 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-02-10T05:35:36.122 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T05:35:36.122 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:35:36.122 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-10T05:35:36.122 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:35:36.122 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:35:36.122 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-02-10 05:34:56.486716340 +0000 2024-02-10T05:35:36.123 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-02-10 05:32:17.222543416 +0000 2024-02-10T05:35:36.123 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-02-10 05:32:17.222543416 +0000 2024-02-10T05:35:36.123 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-02-10 05:32:17.222543416 +0000 2024-02-10T05:35:36.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T05:35:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-02-10T05:35:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-02-10T05:35:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.00010403 s, 4.9 MB/s 2024-02-10T05:35:36.189 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T05:35:36.244 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-02-10T05:35:36.298 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T05:35:36.299 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:35:36.299 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-10T05:35:36.299 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:35:36.299 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:35:36.299 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-02-10 05:34:56.487716314 +0000 2024-02-10T05:35:36.299 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-02-10 05:32:17.556534570 +0000 2024-02-10T05:35:36.299 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-02-10 05:32:17.556534570 +0000 2024-02-10T05:35:36.299 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-02-10 05:32:17.556534570 +0000 2024-02-10T05:35:36.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T05:35:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-02-10T05:35:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-02-10T05:35:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 9.9787e-05 s, 5.1 MB/s 2024-02-10T05:35:36.365 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T05:35:36.420 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-02-10T05:35:36.475 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T05:35:36.475 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:35:36.475 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-10T05:35:36.475 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:35:36.475 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:35:36.475 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-02-10 05:34:56.487716314 +0000 2024-02-10T05:35:36.475 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-02-10 05:32:17.887525802 +0000 2024-02-10T05:35:36.475 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-02-10 05:32:17.887525802 +0000 2024-02-10T05:35:36.475 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-02-10 05:32:17.887525802 +0000 2024-02-10T05:35:36.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T05:35:36.540 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-02-10T05:35:36.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-02-10T05:35:36.541 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 8.4101e-05 s, 6.1 MB/s 2024-02-10T05:35:36.543 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T05:35:36.599 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-02-10T05:35:36.654 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T05:35:36.654 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:35:36.654 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-10T05:35:36.654 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:35:36.654 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:35:36.654 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-02-10 05:34:56.487716314 +0000 2024-02-10T05:35:36.654 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-02-10 05:32:18.226516823 +0000 2024-02-10T05:35:36.654 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-02-10 05:32:18.226516823 +0000 2024-02-10T05:35:36.655 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-02-10 05:32:18.226516823 +0000 2024-02-10T05:35:36.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T05:35:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-02-10T05:35:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-02-10T05:35:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 9.4086e-05 s, 5.4 MB/s 2024-02-10T05:35:36.725 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T05:35:36.780 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-02-10T05:35:36.781 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:35:36.799 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T05:35:36.799 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-02-10T05:35:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T05:35:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:35:36.857 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-10T05:35:36.858 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:35:36.858 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:35:36.858 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-02-10 05:35:31.307178460 +0000 2024-02-10T05:35:36.858 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-02-10 05:32:12.391294604 +0000 2024-02-10T05:35:36.858 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-02-10 05:32:12.391294604 +0000 2024-02-10T05:35:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-02-10 05:32:12.391294604 +0000 2024-02-10T05:35:36.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T05:35:36.927 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-02-10T05:35:36.927 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-02-10T05:35:36.927 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.00022361 s, 2.3 MB/s 2024-02-10T05:35:36.928 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T05:35:36.986 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-02-10T05:35:37.043 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T05:35:37.043 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:35:37.043 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-10T05:35:37.043 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:35:37.043 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:35:37.043 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-02-10 05:35:31.308178439 +0000 2024-02-10T05:35:37.043 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-02-10 05:32:12.920283715 +0000 2024-02-10T05:35:37.043 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-02-10 05:32:12.920283715 +0000 2024-02-10T05:35:37.043 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-02-10 05:32:12.920283715 +0000 2024-02-10T05:35:37.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T05:35:37.112 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-02-10T05:35:37.112 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-02-10T05:35:37.112 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000204048 s, 2.5 MB/s 2024-02-10T05:35:37.114 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T05:35:37.172 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-02-10T05:35:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T05:35:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:35:37.228 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-10T05:35:37.229 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:35:37.229 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:35:37.229 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-02-10 05:35:31.308178439 +0000 2024-02-10T05:35:37.229 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-02-10 05:32:13.273276450 +0000 2024-02-10T05:35:37.229 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-02-10 05:32:13.273276450 +0000 2024-02-10T05:35:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-02-10 05:32:13.273276450 +0000 2024-02-10T05:35:37.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T05:35:37.293 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:36 smithi192 ceph-mon[26904]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:37.293 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:36 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:37.295 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-02-10T05:35:37.295 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-02-10T05:35:37.296 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000181177 s, 2.8 MB/s 2024-02-10T05:35:37.297 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T05:35:37.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:36 smithi028 ceph-mon[24680]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:37.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:36 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:37.354 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-02-10T05:35:37.410 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T05:35:37.410 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:35:37.410 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-10T05:35:37.410 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:35:37.410 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:35:37.410 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-02-10 05:35:31.309178419 +0000 2024-02-10T05:35:37.411 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-02-10 05:32:13.619269328 +0000 2024-02-10T05:35:37.411 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-02-10 05:32:13.619269328 +0000 2024-02-10T05:35:37.411 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-02-10 05:32:13.619269328 +0000 2024-02-10T05:35:37.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T05:35:37.477 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-02-10T05:35:37.477 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-02-10T05:35:37.477 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000221421 s, 2.3 MB/s 2024-02-10T05:35:37.478 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T05:35:37.534 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-02-10T05:35:37.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T05:35:39.235 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:38 smithi028 ceph-mon[24680]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:39.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:38 smithi192 ceph-mon[26904]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:40.084 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T05:35:40.084 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T05:35:40.084 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-02-10T05:35:40.085 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-02-10T05:35:40.085 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.010625 s, 987 MB/s 2024-02-10T05:35:40.085 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-02-10T05:35:41.171 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:40 smithi028 ceph-mon[24680]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:41.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:40 smithi192 ceph-mon[26904]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:41.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-02-10T05:35:43.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:42 smithi028 ceph-mon[24680]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:42 smithi192 ceph-mon[26904]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:45.197 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:44 smithi028 ceph-mon[24680]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:45.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:44 smithi192 ceph-mon[26904]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:46.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:45 smithi028 ceph-mon[24680]: from='client.14234 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:35:46.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:45 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:35:46.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:45 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:35:46.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:45 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:45 smithi192 ceph-mon[26904]: from='client.14234 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:35:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:45 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:35:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:45 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:35:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:45 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:47.042 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:46 smithi028 ceph-mon[24680]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:46 smithi192 ceph-mon[26904]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:49.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:49 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:49.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:49 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:35:49.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:49 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:49.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:49 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:35:49.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:49 smithi028 ceph-mon[24680]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:49.559 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:49 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:49.560 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:49 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:35:49.560 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:49 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:49.560 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:49 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:35:49.560 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:49 smithi192 ceph-mon[26904]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:50.365 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:50 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2384004757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2f2678e-aca7-4091-af12-a513153910b4"}]: dispatch 2024-02-10T05:35:50.365 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:50 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2384004757' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2f2678e-aca7-4091-af12-a513153910b4"}]': finished 2024-02-10T05:35:50.365 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:50 smithi192 ceph-mon[26904]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:35:50.365 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:35:50.365 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:50.476 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:50 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2384004757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2f2678e-aca7-4091-af12-a513153910b4"}]: dispatch 2024-02-10T05:35:50.476 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:50 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2384004757' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2f2678e-aca7-4091-af12-a513153910b4"}]': finished 2024-02-10T05:35:50.476 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:50 smithi028 ceph-mon[24680]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:35:50.476 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:35:50.476 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:51.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:51 smithi028 ceph-mon[24680]: Updating smithi192:/etc/ceph/ceph.conf 2024-02-10T05:35:51.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:51 smithi028 ceph-mon[24680]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:35:51.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:51 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:51.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:51 smithi028 ceph-mon[24680]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:51.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:51 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1119580695' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:35:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:51 smithi192 ceph-mon[26904]: Updating smithi192:/etc/ceph/ceph.conf 2024-02-10T05:35:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:51 smithi192 ceph-mon[26904]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:35:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:51 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:51 smithi192 ceph-mon[26904]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:51 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/1119580695' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:35:53.254 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:52 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:53.254 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:52 smithi028 ceph-mon[24680]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:52 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:52 smithi192 ceph-mon[26904]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:55.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:55.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:54 smithi028 ceph-mon[24680]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-10T05:35:55.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:55.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:55.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:35:55.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:35:55.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:55.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:55.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:54 smithi192 ceph-mon[26904]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-10T05:35:55.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:55.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:55.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:35:55.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:35:55.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:56.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:55 smithi028 ceph-mon[24680]: Reconfiguring mon.smithi028 (unknown last config time)... 2024-02-10T05:35:56.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:55 smithi028 ceph-mon[24680]: Reconfiguring daemon mon.smithi028 on smithi028 2024-02-10T05:35:56.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:55 smithi028 ceph-mon[24680]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:56.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:55 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:56.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:55 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi028.ucuqfx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:35:56.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:55 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:35:56.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:55 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:56.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:55 smithi192 ceph-mon[26904]: Reconfiguring mon.smithi028 (unknown last config time)... 2024-02-10T05:35:56.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:55 smithi192 ceph-mon[26904]: Reconfiguring daemon mon.smithi028 on smithi028 2024-02-10T05:35:56.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:55 smithi192 ceph-mon[26904]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:56.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:55 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:56.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:55 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi028.ucuqfx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:35:56.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:55 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:35:56.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:55 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:35:57.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:56 smithi028 ceph-mon[24680]: Reconfiguring mgr.smithi028.ucuqfx (unknown last config time)... 2024-02-10T05:35:57.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:56 smithi028 ceph-mon[24680]: Reconfiguring daemon mgr.smithi028.ucuqfx on smithi028 2024-02-10T05:35:57.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:56 smithi192 ceph-mon[26904]: Reconfiguring mgr.smithi028.ucuqfx (unknown last config time)... 2024-02-10T05:35:57.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:56 smithi192 ceph-mon[26904]: Reconfiguring daemon mgr.smithi028.ucuqfx on smithi028 2024-02-10T05:35:57.999 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:57 smithi028 ceph-mon[24680]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:57.999 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:57 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:58.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:57 smithi192 ceph-mon[26904]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:58.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:57 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:35:58.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:58 smithi028 ceph-mon[24680]: Reconfiguring alertmanager.smithi028 (dependencies changed)... 2024-02-10T05:35:58.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:58 smithi028 ceph-mon[24680]: Reconfiguring daemon alertmanager.smithi028 on smithi028 2024-02-10T05:35:59.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:58 smithi192 ceph-mon[26904]: Reconfiguring alertmanager.smithi028 (dependencies changed)... 2024-02-10T05:35:59.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:58 smithi192 ceph-mon[26904]: Reconfiguring daemon alertmanager.smithi028 on smithi028 2024-02-10T05:35:59.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:59 smithi028 ceph-mon[24680]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:35:59.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:59 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:35:59.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:35:59 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:00.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:59 smithi192 ceph-mon[26904]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:00.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:59 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:36:00.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:35:59 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:00.791 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:00 smithi028 ceph-mon[24680]: Deploying daemon osd.0 on smithi028 2024-02-10T05:36:01.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:00 smithi192 ceph-mon[26904]: Deploying daemon osd.0 on smithi028 2024-02-10T05:36:01.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:01 smithi028 ceph-mon[24680]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:02.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:01 smithi192 ceph-mon[26904]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:03.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:02 smithi028 ceph-mon[24680]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:03.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:02 smithi192 ceph-mon[26904]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:04.437 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:04.437 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:04 smithi028 ceph-mon[24680]: Reconfiguring crash.smithi028 (monmap changed)... 2024-02-10T05:36:04.437 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi028", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:36:04.437 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:04.437 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:04 smithi028 ceph-mon[24680]: Reconfiguring daemon crash.smithi028 on smithi028 2024-02-10T05:36:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:04 smithi192 ceph-mon[26904]: Reconfiguring crash.smithi028 (monmap changed)... 2024-02-10T05:36:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi028", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:36:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:04 smithi192 ceph-mon[26904]: Reconfiguring daemon crash.smithi028 on smithi028 2024-02-10T05:36:05.326 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:05 smithi028 ceph-mon[24680]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:05.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:05 smithi192 ceph-mon[26904]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:07.184 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:06 smithi028 ceph-mon[24680]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:07.185 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:36:07.185 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:36:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:06 smithi192 ceph-mon[26904]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:36:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:36:09.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:09 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:09.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:09 smithi028 ceph-mon[24680]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:09 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:09 smithi192 ceph-mon[26904]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:10.168 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:10 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:10.168 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:10 smithi028 ceph-mon[24680]: Reconfiguring grafana.smithi028 (dependencies changed)... 2024-02-10T05:36:10.168 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:10 smithi028 ceph-mon[24680]: Reconfiguring daemon grafana.smithi028 on smithi028 2024-02-10T05:36:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:10 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:10 smithi192 ceph-mon[26904]: Reconfiguring grafana.smithi028 (dependencies changed)... 2024-02-10T05:36:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:10 smithi192 ceph-mon[26904]: Reconfiguring daemon grafana.smithi028 on smithi028 2024-02-10T05:36:11.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:11 smithi028 ceph-mon[24680]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:11.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:11 smithi028 ceph-mon[24680]: from='osd.0 [v2:172.21.15.28:6802/951842531,v1:172.21.15.28:6803/951842531]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:36:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:11 smithi192 ceph-mon[26904]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:11 smithi192 ceph-mon[26904]: from='osd.0 [v2:172.21.15.28:6802/951842531,v1:172.21.15.28:6803/951842531]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:36:12.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:12 smithi028 ceph-mon[24680]: from='osd.0 [v2:172.21.15.28:6802/951842531,v1:172.21.15.28:6803/951842531]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:36:12.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:12 smithi028 ceph-mon[24680]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:36:12.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:12 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:36:12.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:12 smithi028 ceph-mon[24680]: from='osd.0 [v2:172.21.15.28:6802/951842531,v1:172.21.15.28:6803/951842531]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:36:12.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:12 smithi192 ceph-mon[26904]: from='osd.0 [v2:172.21.15.28:6802/951842531,v1:172.21.15.28:6803/951842531]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:36:12.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:12 smithi192 ceph-mon[26904]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:36:12.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:12 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:36:12.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:12 smithi192 ceph-mon[26904]: from='osd.0 [v2:172.21.15.28:6802/951842531,v1:172.21.15.28:6803/951842531]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:36:13.437 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:13 smithi028 ceph-mon[24680]: from='osd.0 [v2:172.21.15.28:6802/951842531,v1:172.21.15.28:6803/951842531]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-10T05:36:13.437 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:13 smithi028 ceph-mon[24680]: osdmap e8: 1 total, 0 up, 1 in 2024-02-10T05:36:13.438 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:13 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:36:13.438 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:13 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:36:13.438 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:13 smithi028 ceph-mon[24680]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:13 smithi192 ceph-mon[26904]: from='osd.0 [v2:172.21.15.28:6802/951842531,v1:172.21.15.28:6803/951842531]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-10T05:36:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:13 smithi192 ceph-mon[26904]: osdmap e8: 1 total, 0 up, 1 in 2024-02-10T05:36:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:13 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:36:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:13 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:36:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:13 smithi192 ceph-mon[26904]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:36:14.205 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:14 smithi028 ceph-mon[24680]: purged_snaps scrub starts 2024-02-10T05:36:14.205 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:14 smithi028 ceph-mon[24680]: purged_snaps scrub ok 2024-02-10T05:36:14.205 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:14 smithi028 ceph-mon[24680]: osd.0 [v2:172.21.15.28:6802/951842531,v1:172.21.15.28:6803/951842531] boot 2024-02-10T05:36:14.205 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:14 smithi028 ceph-mon[24680]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:36:14.205 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:36:14.205 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:14.206 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:14 smithi028 ceph-mon[24680]: Reconfiguring crash.smithi192 (monmap changed)... 2024-02-10T05:36:14.206 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:36:14.206 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:14.206 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:14 smithi028 ceph-mon[24680]: Reconfiguring daemon crash.smithi192 on smithi192 2024-02-10T05:36:14.417 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:14 smithi192 ceph-mon[26904]: purged_snaps scrub starts 2024-02-10T05:36:14.417 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:14 smithi192 ceph-mon[26904]: purged_snaps scrub ok 2024-02-10T05:36:14.417 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:14 smithi192 ceph-mon[26904]: osd.0 [v2:172.21.15.28:6802/951842531,v1:172.21.15.28:6803/951842531] boot 2024-02-10T05:36:14.417 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:14 smithi192 ceph-mon[26904]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:36:14.418 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:36:14.418 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:14.418 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:14 smithi192 ceph-mon[26904]: Reconfiguring crash.smithi192 (monmap changed)... 2024-02-10T05:36:14.418 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:36:14.418 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:14.418 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:14 smithi192 ceph-mon[26904]: Reconfiguring daemon crash.smithi192 on smithi192 2024-02-10T05:36:15.292 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-02-10T05:36:16.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:15 smithi028 ceph-mon[24680]: pgmap v31: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:16.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:15 smithi028 ceph-mon[24680]: osdmap e10: 1 total, 1 up, 1 in 2024-02-10T05:36:16.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:15 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:16.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:15 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.hppzcm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:36:16.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:15 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:36:16.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:15 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:16.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:15 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:16.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:15 smithi192 ceph-mon[26904]: pgmap v31: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:16.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:15 smithi192 ceph-mon[26904]: osdmap e10: 1 total, 1 up, 1 in 2024-02-10T05:36:16.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:15 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:16.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:15 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.hppzcm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:36:16.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:15 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:36:16.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:15 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:16.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:15 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:16.137 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.0.service 2024-02-10T05:36:16.139 INFO:tasks.cephadm:Deploying osd.1 on smithi028 with /dev/vg_nvme/lv_3... 2024-02-10T05:36:16.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T05:36:17.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:16 smithi028 ceph-mon[24680]: Reconfiguring mgr.smithi192.hppzcm (monmap changed)... 2024-02-10T05:36:17.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:16 smithi028 ceph-mon[24680]: Reconfiguring daemon mgr.smithi192.hppzcm on smithi192 2024-02-10T05:36:17.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:16 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:17.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:16 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:36:17.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:16 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:36:17.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:16 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:17.085 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:16 smithi192 ceph-mon[26904]: Reconfiguring mgr.smithi192.hppzcm (monmap changed)... 2024-02-10T05:36:17.085 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:16 smithi192 ceph-mon[26904]: Reconfiguring daemon mgr.smithi192.hppzcm on smithi192 2024-02-10T05:36:17.085 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:16 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:17.085 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:16 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:36:17.085 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:16 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:36:17.085 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:16 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:17.996 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:17 smithi028 ceph-mon[24680]: Reconfiguring mon.smithi192 (monmap changed)... 2024-02-10T05:36:17.996 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:17 smithi028 ceph-mon[24680]: Reconfiguring daemon mon.smithi192 on smithi192 2024-02-10T05:36:17.996 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:17 smithi028 ceph-mon[24680]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T05:36:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T05:36:18.213 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-02-10T05:36:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-02-10T05:36:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0102459 s, 1.0 GB/s 2024-02-10T05:36:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-02-10T05:36:18.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:17 smithi192 ceph-mon[26904]: Reconfiguring mon.smithi192 (monmap changed)... 2024-02-10T05:36:18.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:17 smithi192 ceph-mon[26904]: Reconfiguring daemon mon.smithi192 on smithi192 2024-02-10T05:36:18.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:17 smithi192 ceph-mon[26904]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:18.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-02-10T05:36:19.176 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:19.176 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:36:19.176 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.28:9093"}]: dispatch 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.28:9093"}]: dispatch 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.28:3000"}]: dispatch 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.28:3000"}]: dispatch 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.28:9095"}]: dispatch 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.28:9095"}]: dispatch 2024-02-10T05:36:19.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:19.178 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:36:19.178 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:19.178 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:36:19.178 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:18 smithi192 ceph-mon[26904]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:19.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:19.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:36:19.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:36:19.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.28:9093"}]: dispatch 2024-02-10T05:36:19.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.28:9093"}]: dispatch 2024-02-10T05:36:19.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:19.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:36:19.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:36:19.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.28:3000"}]: dispatch 2024-02-10T05:36:19.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.28:3000"}]: dispatch 2024-02-10T05:36:19.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:19.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:36:19.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:36:19.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.28:9095"}]: dispatch 2024-02-10T05:36:19.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.28:9095"}]: dispatch 2024-02-10T05:36:19.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:19.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:36:19.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:19.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:36:19.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:18 smithi028 ceph-mon[24680]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:20.851 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:20 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:20 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:21.945 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:21 smithi028 ceph-mon[24680]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:22.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:21 smithi192 ceph-mon[26904]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:22.804 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:22 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:36:22.804 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:22 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:36:22.804 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:22 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:23.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:22 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:36:23.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:22 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:36:23.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:22 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:23.935 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:23 smithi028 ceph-mon[24680]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:36:23.935 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:23 smithi028 ceph-mon[24680]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:23.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:23 smithi192 ceph-mon[26904]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:36:23.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:23 smithi192 ceph-mon[26904]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:25.130 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:24 smithi192 ceph-mon[26904]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:25.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:24 smithi028 ceph-mon[24680]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:26.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:25 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3425458813' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c619156b-7b30-4239-9ccf-84de21b54db3"}]: dispatch 2024-02-10T05:36:26.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:25 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3425458813' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c619156b-7b30-4239-9ccf-84de21b54db3"}]': finished 2024-02-10T05:36:26.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:25 smithi028 ceph-mon[24680]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:36:26.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:25 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:36:26.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:25 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:26.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:25 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1047295252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:36:26.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/3425458813' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c619156b-7b30-4239-9ccf-84de21b54db3"}]: dispatch 2024-02-10T05:36:26.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/3425458813' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c619156b-7b30-4239-9ccf-84de21b54db3"}]': finished 2024-02-10T05:36:26.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:25 smithi192 ceph-mon[26904]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:36:26.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:36:26.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:25 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:26.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:25 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/1047295252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:36:27.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:26 smithi028 ceph-mon[24680]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:26 smithi192 ceph-mon[26904]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:29.033 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:28 smithi028 ceph-mon[24680]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:28 smithi192 ceph-mon[26904]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:31 smithi192 ceph-mon[26904]: Detected new or changed devices on smithi028 2024-02-10T05:36:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:31 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:31 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:36:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:31 smithi192 ceph-mon[26904]: Adjusting osd_memory_target on smithi028 to 12869M 2024-02-10T05:36:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:31 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:31 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:31 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:36:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:31 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:31 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:36:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:31 smithi192 ceph-mon[26904]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:31.543 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:31 smithi028 ceph-mon[24680]: Detected new or changed devices on smithi028 2024-02-10T05:36:31.543 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:31 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:31.543 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:31 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:36:31.543 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:31 smithi028 ceph-mon[24680]: Adjusting osd_memory_target on smithi028 to 12869M 2024-02-10T05:36:31.543 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:31 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:31.543 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:31 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:31.543 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:31 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:36:31.543 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:31 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:31.543 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:31 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:36:31.543 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:31 smithi028 ceph-mon[24680]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:33.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:32 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:33.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:32 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:36:33.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:32 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:33.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:32 smithi028 ceph-mon[24680]: Deploying daemon osd.1 on smithi028 2024-02-10T05:36:33.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:32 smithi028 ceph-mon[24680]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:32 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:32 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:36:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:32 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:32 smithi192 ceph-mon[26904]: Deploying daemon osd.1 on smithi028 2024-02-10T05:36:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:32 smithi192 ceph-mon[26904]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:35.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:34 smithi028 ceph-mon[24680]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:35.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:34 smithi192 ceph-mon[26904]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:37.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:36 smithi028 ceph-mon[24680]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:36 smithi192 ceph-mon[26904]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:38.333 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:38 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:38.333 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:38 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:38.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:38 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:38.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:38 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:39.491 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:39 smithi028 ceph-mon[24680]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:39.491 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:39 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:39.491 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:39 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:36:39.491 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:39 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:39.491 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:39 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:36:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:39 smithi192 ceph-mon[26904]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:39 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:39 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:36:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:39 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:39 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:36:42.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:41 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:42.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:41 smithi028 ceph-mon[24680]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:42.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:41 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:42.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:41 smithi192 ceph-mon[26904]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:42.251 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 1 on host 'smithi028' 2024-02-10T05:36:42.814 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:42 smithi028 ceph-mon[24680]: from='osd.1 [v2:172.21.15.28:6810/2828860592,v1:172.21.15.28:6811/2828860592]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:36:42.814 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:42 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:42.842 DEBUG:teuthology.orchestra.run.smithi028:osd.1> sudo journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.1.service 2024-02-10T05:36:42.845 INFO:tasks.cephadm:Deploying osd.2 on smithi028 with /dev/vg_nvme/lv_2... 2024-02-10T05:36:42.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T05:36:43.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:42 smithi192 ceph-mon[26904]: from='osd.1 [v2:172.21.15.28:6810/2828860592,v1:172.21.15.28:6811/2828860592]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:36:43.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:42 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:44.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:43 smithi192 ceph-mon[26904]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:44.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:43 smithi192 ceph-mon[26904]: from='osd.1 [v2:172.21.15.28:6810/2828860592,v1:172.21.15.28:6811/2828860592]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:36:44.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:43 smithi192 ceph-mon[26904]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:36:44.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:36:44.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:43 smithi192 ceph-mon[26904]: from='osd.1 [v2:172.21.15.28:6810/2828860592,v1:172.21.15.28:6811/2828860592]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:36:44.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:43 smithi028 ceph-mon[24680]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:44.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:43 smithi028 ceph-mon[24680]: from='osd.1 [v2:172.21.15.28:6810/2828860592,v1:172.21.15.28:6811/2828860592]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:36:44.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:43 smithi028 ceph-mon[24680]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:36:44.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:36:44.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:43 smithi028 ceph-mon[24680]: from='osd.1 [v2:172.21.15.28:6810/2828860592,v1:172.21.15.28:6811/2828860592]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:36:44.309 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:36:43 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[39144]: 2024-02-10T05:36:43.811+0000 7f6edc526700 -1 osd.1 0 waiting for initial osdmap 2024-02-10T05:36:44.310 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:36:43 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[39144]: 2024-02-10T05:36:43.817+0000 7f6ed4eb9700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:36:44.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:44 smithi028 ceph-mon[24680]: from='osd.1 [v2:172.21.15.28:6810/2828860592,v1:172.21.15.28:6811/2828860592]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-10T05:36:44.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:44 smithi028 ceph-mon[24680]: osdmap e13: 2 total, 1 up, 2 in 2024-02-10T05:36:44.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:44 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:36:44.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:44 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:36:45.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:44 smithi192 ceph-mon[26904]: from='osd.1 [v2:172.21.15.28:6810/2828860592,v1:172.21.15.28:6811/2828860592]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-10T05:36:45.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:44 smithi192 ceph-mon[26904]: osdmap e13: 2 total, 1 up, 2 in 2024-02-10T05:36:45.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:44 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:36:45.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:44 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:36:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T05:36:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T05:36:45.463 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-02-10T05:36:45.464 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-02-10T05:36:45.464 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0105416 s, 995 MB/s 2024-02-10T05:36:45.464 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-02-10T05:36:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:45 smithi192 ceph-mon[26904]: purged_snaps scrub starts 2024-02-10T05:36:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:45 smithi192 ceph-mon[26904]: purged_snaps scrub ok 2024-02-10T05:36:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:45 smithi192 ceph-mon[26904]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:45 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:36:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:45 smithi192 ceph-mon[26904]: osd.1 [v2:172.21.15.28:6810/2828860592,v1:172.21.15.28:6811/2828860592] boot 2024-02-10T05:36:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:45 smithi192 ceph-mon[26904]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:36:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:45 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:36:46.115 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:45 smithi028 ceph-mon[24680]: purged_snaps scrub starts 2024-02-10T05:36:46.115 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:45 smithi028 ceph-mon[24680]: purged_snaps scrub ok 2024-02-10T05:36:46.115 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:45 smithi028 ceph-mon[24680]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:36:46.116 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:45 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:36:46.116 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:45 smithi028 ceph-mon[24680]: osd.1 [v2:172.21.15.28:6810/2828860592,v1:172.21.15.28:6811/2828860592] boot 2024-02-10T05:36:46.116 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:45 smithi028 ceph-mon[24680]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:36:46.116 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:45 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:36:46.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-02-10T05:36:47.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:46 smithi028 ceph-mon[24680]: osdmap e15: 2 total, 2 up, 2 in 2024-02-10T05:36:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:46 smithi192 ceph-mon[26904]: osdmap e15: 2 total, 2 up, 2 in 2024-02-10T05:36:48.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:47 smithi028 ceph-mon[24680]: pgmap v53: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:48.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:47 smithi192 ceph-mon[26904]: pgmap v53: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:49.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:48 smithi028 ceph-mon[24680]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:48 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:36:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:48 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:36:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:48 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:48 smithi192 ceph-mon[26904]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:48 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:36:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:48 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:36:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:48 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:50.717 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:50 smithi028 ceph-mon[24680]: from='client.14270 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:36:50.718 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:50 smithi192 ceph-mon[26904]: from='client.14270 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:36:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:51.734 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:51 smithi028 ceph-mon[24680]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:51.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:51 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3425472055' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80df6130-a9d6-4098-a828-3bebb33d2621"}]: dispatch 2024-02-10T05:36:51.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:51 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3425472055' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80df6130-a9d6-4098-a828-3bebb33d2621"}]': finished 2024-02-10T05:36:51.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:51 smithi028 ceph-mon[24680]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:36:51.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:51 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:36:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:51 smithi192 ceph-mon[26904]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:51 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/3425472055' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80df6130-a9d6-4098-a828-3bebb33d2621"}]: dispatch 2024-02-10T05:36:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:51 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/3425472055' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80df6130-a9d6-4098-a828-3bebb33d2621"}]': finished 2024-02-10T05:36:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:51 smithi192 ceph-mon[26904]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:36:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:51 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:36:52.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:52 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2282036959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:36:52.821 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:52 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2282036959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:36:53.741 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:53 smithi192 ceph-mon[26904]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:53.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:53 smithi028 ceph-mon[24680]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:55.138 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:54 smithi028 ceph-mon[24680]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:55.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:54 smithi192 ceph-mon[26904]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:57.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:56 smithi028 ceph-mon[24680]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:57.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:56 smithi192 ceph-mon[26904]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:58.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:58 smithi028 ceph-mon[24680]: Detected new or changed devices on smithi028 2024-02-10T05:36:58.736 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:58 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:58.736 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:58 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:36:58.736 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:58 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:36:58.736 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:58 smithi028 ceph-mon[24680]: Adjusting osd_memory_target on smithi028 to 6434M 2024-02-10T05:36:58.736 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:58 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:58.736 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:58 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:58.736 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:58 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:36:58.736 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:58 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:58.736 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:58 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:36:59.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:58 smithi192 ceph-mon[26904]: Detected new or changed devices on smithi028 2024-02-10T05:36:59.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:58 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:59.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:58 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:36:59.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:58 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:36:59.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:58 smithi192 ceph-mon[26904]: Adjusting osd_memory_target on smithi028 to 6434M 2024-02-10T05:36:59.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:58 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:59.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:58 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:36:59.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:58 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:36:59.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:58 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:36:59.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:58 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:36:59.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:59 smithi028 ceph-mon[24680]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:36:59.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:59 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:36:59.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:36:59 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:00.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:59 smithi192 ceph-mon[26904]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:00.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:59 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:37:00.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:36:59 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:01.005 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:00 smithi028 ceph-mon[24680]: Deploying daemon osd.2 on smithi028 2024-02-10T05:37:01.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:00 smithi192 ceph-mon[26904]: Deploying daemon osd.2 on smithi028 2024-02-10T05:37:01.713 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:01 smithi028 ceph-mon[24680]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:02.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:01 smithi192 ceph-mon[26904]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:03.182 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:02 smithi028 ceph-mon[24680]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:03.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:02 smithi192 ceph-mon[26904]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:05.201 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:04 smithi028 ceph-mon[24680]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:05.201 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:05.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:04 smithi192 ceph-mon[26904]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:05.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:06.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:05 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:06.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:05 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:06 smithi028 ceph-mon[24680]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:37:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:37:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:37:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:37:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:06 smithi192 ceph-mon[26904]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:37:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:37:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:37:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:37:09.077 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:08 smithi028 ceph-mon[24680]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:08 smithi192 ceph-mon[26904]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:09.907 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 2 on host 'smithi028' 2024-02-10T05:37:10.763 DEBUG:teuthology.orchestra.run.smithi028:osd.2> sudo journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.2.service 2024-02-10T05:37:10.766 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_1... 2024-02-10T05:37:10.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T05:37:11.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:10 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:11.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:10 smithi028 ceph-mon[24680]: from='osd.2 [v2:172.21.15.28:6818/1602733598,v1:172.21.15.28:6819/1602733598]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:37:11.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:10 smithi028 ceph-mon[24680]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:11.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:10 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:11.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:10 smithi192 ceph-mon[26904]: from='osd.2 [v2:172.21.15.28:6818/1602733598,v1:172.21.15.28:6819/1602733598]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:37:11.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:10 smithi192 ceph-mon[26904]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:12.206 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:11 smithi028 ceph-mon[24680]: from='osd.2 [v2:172.21.15.28:6818/1602733598,v1:172.21.15.28:6819/1602733598]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:37:12.206 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:11 smithi028 ceph-mon[24680]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:37:12.206 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:11 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:37:12.206 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:11 smithi028 ceph-mon[24680]: from='osd.2 [v2:172.21.15.28:6818/1602733598,v1:172.21.15.28:6819/1602733598]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:37:12.207 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:37:11 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2[42852]: 2024-02-10T05:37:11.913+0000 7f0073048700 -1 osd.2 0 waiting for initial osdmap 2024-02-10T05:37:12.207 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:37:11 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2[42852]: 2024-02-10T05:37:11.917+0000 7f006d1de700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:37:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:11 smithi192 ceph-mon[26904]: from='osd.2 [v2:172.21.15.28:6818/1602733598,v1:172.21.15.28:6819/1602733598]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:37:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:11 smithi192 ceph-mon[26904]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:37:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:11 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:37:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:11 smithi192 ceph-mon[26904]: from='osd.2 [v2:172.21.15.28:6818/1602733598,v1:172.21.15.28:6819/1602733598]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:37:12.945 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T05:37:12.945 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T05:37:12.945 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-02-10T05:37:12.945 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-02-10T05:37:12.946 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130969 s, 801 MB/s 2024-02-10T05:37:12.946 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-02-10T05:37:13.078 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:12 smithi028 ceph-mon[24680]: from='osd.2 [v2:172.21.15.28:6818/1602733598,v1:172.21.15.28:6819/1602733598]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-10T05:37:13.078 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:12 smithi028 ceph-mon[24680]: osdmap e18: 3 total, 2 up, 3 in 2024-02-10T05:37:13.078 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:12 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:37:13.079 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:12 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:37:13.079 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:12 smithi028 ceph-mon[24680]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:13.079 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:12 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:37:13.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:12 smithi192 ceph-mon[26904]: from='osd.2 [v2:172.21.15.28:6818/1602733598,v1:172.21.15.28:6819/1602733598]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-10T05:37:13.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:12 smithi192 ceph-mon[26904]: osdmap e18: 3 total, 2 up, 3 in 2024-02-10T05:37:13.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:12 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:37:13.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:12 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:37:13.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:12 smithi192 ceph-mon[26904]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:37:13.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:12 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:37:13.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-02-10T05:37:14.204 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:13 smithi028 ceph-mon[24680]: purged_snaps scrub starts 2024-02-10T05:37:14.204 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:13 smithi028 ceph-mon[24680]: purged_snaps scrub ok 2024-02-10T05:37:14.204 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:13 smithi028 ceph-mon[24680]: osd.2 [v2:172.21.15.28:6818/1602733598,v1:172.21.15.28:6819/1602733598] boot 2024-02-10T05:37:14.204 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:13 smithi028 ceph-mon[24680]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:37:14.205 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:13 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:37:14.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:13 smithi192 ceph-mon[26904]: purged_snaps scrub starts 2024-02-10T05:37:14.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:13 smithi192 ceph-mon[26904]: purged_snaps scrub ok 2024-02-10T05:37:14.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:13 smithi192 ceph-mon[26904]: osd.2 [v2:172.21.15.28:6818/1602733598,v1:172.21.15.28:6819/1602733598] boot 2024-02-10T05:37:14.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:13 smithi192 ceph-mon[26904]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:37:14.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:13 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:37:15.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:15 smithi028 ceph-mon[24680]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:37:15.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:15 smithi028 ceph-mon[24680]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:15 smithi192 ceph-mon[26904]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:37:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:15 smithi192 ceph-mon[26904]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:16.379 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:16 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T05:37:16.380 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:16 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T05:37:16.380 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:16 smithi028 ceph-mon[24680]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:37:16.380 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:16 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:37:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:16 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-10T05:37:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:16 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-10T05:37:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:16 smithi192 ceph-mon[26904]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:37:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:16 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:37:17.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:17 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:37:17.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:17 smithi028 ceph-mon[24680]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T05:37:17.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:17 smithi028 ceph-mon[24680]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:17.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:17 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:17.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:17 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:37:17.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:17 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:37:17.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:17 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:17 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:37:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:17 smithi192 ceph-mon[26904]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T05:37:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:17 smithi192 ceph-mon[26904]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:17 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:17 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:37:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:17 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:37:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:17 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:17.744 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:37:17 smithi028 sudo[44432]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:37:17.745 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:37:17 smithi028 sudo[44432]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:37:17.745 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:37:17 smithi028 sudo[44432]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:37:17.745 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:37:17 smithi028 sudo[44455]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:37:17.745 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:37:17 smithi028 sudo[44455]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:37:18.023 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:37:17 smithi028 sudo[44455]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:37:18.292 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:37:18 smithi028 sudo[44471]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:37:18.292 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:37:18 smithi028 sudo[44471]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:37:18.292 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:37:18 smithi028 sudo[44471]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:37:18.548 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:37:18 smithi028 sudo[44488]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:37:18.548 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:37:18 smithi028 sudo[44488]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:37:18.548 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:37:18 smithi028 sudo[44488]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:37:18.808 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:37:18 smithi028 sudo[44496]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:37:18.809 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:37:18 smithi028 sudo[44496]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:37:18.809 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:37:18 smithi028 sudo[44496]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:37:18.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:18 smithi028 ceph-mon[24680]: from='client.14288 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:37:18.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:18.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:18 smithi028 ceph-mon[24680]: osdmap e23: 3 total, 3 up, 3 in 2024-02-10T05:37:19.071 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:37:18 smithi028 sudo[44512]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:37:19.071 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:37:18 smithi028 sudo[44512]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:37:19.071 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:37:18 smithi028 sudo[44512]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:37:19.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:18 smithi192 ceph-mon[26904]: from='client.14288 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:37:19.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:19.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:18 smithi192 ceph-mon[26904]: osdmap e23: 3 total, 3 up, 3 in 2024-02-10T05:37:19.338 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 sudo[44518]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:37:19.338 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 sudo[44518]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:37:19.338 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 sudo[44518]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:37:19.598 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 sudo[44522]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:37:19.598 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 sudo[44522]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:37:19.598 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 sudo[44522]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:37:19.834 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 sudo[29132]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:37:19.834 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 sudo[29132]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:37:19.834 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 sudo[29132]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:37:20.005 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 ceph-mon[24680]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:20.006 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 ceph-mon[24680]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:37:20.006 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 ceph-mon[24680]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:37:20.006 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:37:20.006 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:37:20.006 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 ceph-mon[24680]: mgrmap e19: smithi028.ucuqfx(active, since 2m), standbys: smithi192.hppzcm 2024-02-10T05:37:20.006 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:37:20.006 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:19 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:37:20.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 sudo[29135]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:37:20.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 sudo[29135]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:37:20.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 sudo[29135]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:37:20.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 ceph-mon[26904]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:20.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 ceph-mon[26904]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:37:20.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 ceph-mon[26904]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:37:20.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:37:20.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:37:20.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 ceph-mon[26904]: mgrmap e19: smithi028.ucuqfx(active, since 2m), standbys: smithi192.hppzcm 2024-02-10T05:37:20.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:37:20.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:19 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:37:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:20 smithi192 ceph-mon[26904]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:37:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:20 smithi192 ceph-mon[26904]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:37:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:20 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:20 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/3421288132' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef46ae3c-2805-410e-ac42-e96f69da62c3"}]: dispatch 2024-02-10T05:37:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:20 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/3421288132' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef46ae3c-2805-410e-ac42-e96f69da62c3"}]': finished 2024-02-10T05:37:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:20 smithi192 ceph-mon[26904]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T05:37:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:20 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:37:21.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:20 smithi192 ceph-mon[26904]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:21.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:20 smithi028 ceph-mon[24680]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:37:21.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:20 smithi028 ceph-mon[24680]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:37:21.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:20 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:21.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:20 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3421288132' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef46ae3c-2805-410e-ac42-e96f69da62c3"}]: dispatch 2024-02-10T05:37:21.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:20 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3421288132' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef46ae3c-2805-410e-ac42-e96f69da62c3"}]': finished 2024-02-10T05:37:21.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:20 smithi028 ceph-mon[24680]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T05:37:21.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:20 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:37:21.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:20 smithi028 ceph-mon[24680]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:22.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:21 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2392760153' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:37:22.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:21 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2392760153' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:37:23.160 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:22 smithi028 ceph-mon[24680]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:23.223 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:22 smithi192 ceph-mon[26904]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:25.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:24 smithi028 ceph-mon[24680]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:24 smithi192 ceph-mon[26904]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:27.002 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:26 smithi028 ceph-mon[24680]: Detected new or changed devices on smithi028 2024-02-10T05:37:27.002 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:26 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:27.002 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:26 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:27.003 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:26 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:27.003 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:26 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:27.003 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:26 smithi028 ceph-mon[24680]: Adjusting osd_memory_target on smithi028 to 4289M 2024-02-10T05:37:27.003 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:26 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:27.003 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:26 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:27.003 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:26 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:37:27.003 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:26 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:27.003 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:26 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:37:27.003 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:26 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:27.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:26 smithi192 ceph-mon[26904]: Detected new or changed devices on smithi028 2024-02-10T05:37:27.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:26 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:27.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:26 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:27.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:26 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:27.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:26 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:27.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:26 smithi192 ceph-mon[26904]: Adjusting osd_memory_target on smithi028 to 4289M 2024-02-10T05:37:27.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:26 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:27.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:26 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:27.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:26 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:37:27.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:26 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:27.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:26 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:37:27.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:26 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:27.852 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:27 smithi028 ceph-mon[24680]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:27.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:27 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:37:27.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:27 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:27 smithi192 ceph-mon[26904]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:27 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:37:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:27 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:28.837 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:28 smithi028 ceph-mon[24680]: Deploying daemon osd.3 on smithi028 2024-02-10T05:37:29.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:28 smithi192 ceph-mon[26904]: Deploying daemon osd.3 on smithi028 2024-02-10T05:37:29.814 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:29 smithi028 ceph-mon[24680]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:30.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:29 smithi192 ceph-mon[26904]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:30 smithi192 ceph-mon[26904]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:31.140 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:30 smithi028 ceph-mon[24680]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:33.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:33 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:33.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:33 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:37:33.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:33 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:33.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:33 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:37:33.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:33 smithi028 ceph-mon[24680]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:33.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:33 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:33.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:33 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:37:33.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:33 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:33.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:33 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:37:33.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:33 smithi192 ceph-mon[26904]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:35.023 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:34 smithi028 ceph-mon[24680]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:35.056 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-02-10T05:37:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:34 smithi192 ceph-mon[26904]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:36.053 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.3.service 2024-02-10T05:37:36.055 INFO:tasks.cephadm:Deploying osd.4 on smithi192 with /dev/vg_nvme/lv_4... 2024-02-10T05:37:36.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T05:37:36.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:36 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:36.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:36 smithi028 ceph-mon[24680]: from='osd.3 [v2:172.21.15.28:6826/431349139,v1:172.21.15.28:6827/431349139]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:37:36.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:36 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:36.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:36 smithi192 ceph-mon[26904]: from='osd.3 [v2:172.21.15.28:6826/431349139,v1:172.21.15.28:6827/431349139]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:37:37.231 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:37 smithi028 ceph-mon[24680]: from='osd.3 [v2:172.21.15.28:6826/431349139,v1:172.21.15.28:6827/431349139]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:37:37.231 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:37 smithi028 ceph-mon[24680]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T05:37:37.232 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:37 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:37:37.232 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:37 smithi028 ceph-mon[24680]: from='osd.3 [v2:172.21.15.28:6826/431349139,v1:172.21.15.28:6827/431349139]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:37:37.232 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:37 smithi028 ceph-mon[24680]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:37.232 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:37 smithi028 ceph-mon[24680]: from='osd.3 [v2:172.21.15.28:6826/431349139,v1:172.21.15.28:6827/431349139]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-10T05:37:37.232 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:37:37 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3[46179]: 2024-02-10T05:37:37.061+0000 7fe127d16700 -1 osd.3 0 waiting for initial osdmap 2024-02-10T05:37:37.232 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:37:37 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3[46179]: 2024-02-10T05:37:37.066+0000 7fe120eaa700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:37:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:37 smithi192 ceph-mon[26904]: from='osd.3 [v2:172.21.15.28:6826/431349139,v1:172.21.15.28:6827/431349139]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:37:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:37 smithi192 ceph-mon[26904]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T05:37:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:37 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:37:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:37 smithi192 ceph-mon[26904]: from='osd.3 [v2:172.21.15.28:6826/431349139,v1:172.21.15.28:6827/431349139]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:37:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:37 smithi192 ceph-mon[26904]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:37 smithi192 ceph-mon[26904]: from='osd.3 [v2:172.21.15.28:6826/431349139,v1:172.21.15.28:6827/431349139]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-10T05:37:37.986 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T05:37:37.986 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T05:37:37.987 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-02-10T05:37:37.987 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-02-10T05:37:37.987 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114859 s, 913 MB/s 2024-02-10T05:37:37.987 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-02-10T05:37:38.268 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:38 smithi192 ceph-mon[26904]: osdmap e26: 4 total, 3 up, 4 in 2024-02-10T05:37:38.268 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:38 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:37:38.268 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:38 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:37:38.269 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:38 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:37:38.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:38 smithi028 ceph-mon[24680]: osdmap e26: 4 total, 3 up, 4 in 2024-02-10T05:37:38.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:38 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:37:38.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:38 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:37:38.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:38 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:37:38.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2024-02-10T05:37:39.192 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:39 smithi192 ceph-mon[26904]: purged_snaps scrub starts 2024-02-10T05:37:39.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:39 smithi192 ceph-mon[26904]: purged_snaps scrub ok 2024-02-10T05:37:39.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:39 smithi192 ceph-mon[26904]: osd.3 [v2:172.21.15.28:6826/431349139,v1:172.21.15.28:6827/431349139] boot 2024-02-10T05:37:39.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:39 smithi192 ceph-mon[26904]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T05:37:39.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:39 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:37:39.193 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:39 smithi192 ceph-mon[26904]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:39.208 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:39 smithi028 ceph-mon[24680]: purged_snaps scrub starts 2024-02-10T05:37:39.208 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:39 smithi028 ceph-mon[24680]: purged_snaps scrub ok 2024-02-10T05:37:39.208 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:39 smithi028 ceph-mon[24680]: osd.3 [v2:172.21.15.28:6826/431349139,v1:172.21.15.28:6827/431349139] boot 2024-02-10T05:37:39.209 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:39 smithi028 ceph-mon[24680]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T05:37:39.209 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:39 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:37:39.209 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:39 smithi028 ceph-mon[24680]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:37:40.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:40 smithi028 ceph-mon[24680]: osdmap e28: 4 total, 4 up, 4 in 2024-02-10T05:37:40.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:40 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:40.515 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:40 smithi192 ceph-mon[26904]: osdmap e28: 4 total, 4 up, 4 in 2024-02-10T05:37:40.515 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:40 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:41.168 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:41 smithi192 ceph-mon[26904]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:37:41.169 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:41 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:37:41.169 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:41 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:37:41.169 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:41 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:41.169 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:41 smithi192 ceph-mon[26904]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:41.547 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:41 smithi028 ceph-mon[24680]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:37:41.547 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:41 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:37:41.547 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:41 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:37:41.547 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:41 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:41.547 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:41 smithi028 ceph-mon[24680]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:43.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:43.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:43.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:43.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:43.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:43.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: Adjusting osd_memory_target on smithi028 to 3217M 2024-02-10T05:37:43.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:43.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:43.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:37:43.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:43.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:37:43.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:43.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:43.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/722480524' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c916733-e4c8-4ba7-9995-2cdff4590c93"}]: dispatch 2024-02-10T05:37:43.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c916733-e4c8-4ba7-9995-2cdff4590c93"}]: dispatch 2024-02-10T05:37:43.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c916733-e4c8-4ba7-9995-2cdff4590c93"}]': finished 2024-02-10T05:37:43.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T05:37:43.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:37:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:37:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: Adjusting osd_memory_target on smithi028 to 3217M 2024-02-10T05:37:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:37:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:37:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/722480524' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c916733-e4c8-4ba7-9995-2cdff4590c93"}]: dispatch 2024-02-10T05:37:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c916733-e4c8-4ba7-9995-2cdff4590c93"}]: dispatch 2024-02-10T05:37:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c916733-e4c8-4ba7-9995-2cdff4590c93"}]': finished 2024-02-10T05:37:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T05:37:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:37:44.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:44 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/2808016619' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:37:44.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:44 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/2808016619' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:37:45.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:45 smithi028 ceph-mon[24680]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:45.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:45 smithi192 ceph-mon[26904]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:47.036 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:46 smithi192 ceph-mon[26904]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:47.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:46 smithi028 ceph-mon[24680]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:49.274 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:48 smithi192 ceph-mon[26904]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:49.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:48 smithi028 ceph-mon[24680]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:50.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:49 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:37:50.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:49 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:50.195 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:49 smithi192 ceph-mon[26904]: Deploying daemon osd.4 on smithi192 2024-02-10T05:37:50.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:49 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:37:50.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:49 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:50.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:49 smithi028 ceph-mon[24680]: Deploying daemon osd.4 on smithi192 2024-02-10T05:37:51.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:50 smithi192 ceph-mon[26904]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:50 smithi028 ceph-mon[24680]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:53.017 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:52 smithi192 ceph-mon[26904]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:53.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:52 smithi028 ceph-mon[24680]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:55.196 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:55.197 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:37:55.197 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:55.197 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:37:55.197 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:54 smithi192 ceph-mon[26904]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:55.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:55.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:37:55.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:37:55.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:37:55.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:54 smithi028 ceph-mon[24680]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:57.251 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:56 smithi192 ceph-mon[26904]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:57.281 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 4 on host 'smithi192' 2024-02-10T05:37:57.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:56 smithi028 ceph-mon[24680]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:58.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:57 smithi192 ceph-mon[26904]: from='osd.4 [v2:172.21.15.192:6800/2094940763,v1:172.21.15.192:6801/2094940763]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:37:58.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:57 smithi192 ceph-mon[26904]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:37:58.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:57 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:58.198 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:57 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:58.223 DEBUG:teuthology.orchestra.run.smithi192:osd.4> sudo journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.4.service 2024-02-10T05:37:58.226 INFO:tasks.cephadm:Deploying osd.5 on smithi192 with /dev/vg_nvme/lv_3... 2024-02-10T05:37:58.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T05:37:58.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:57 smithi028 ceph-mon[24680]: from='osd.4 [v2:172.21.15.192:6800/2094940763,v1:172.21.15.192:6801/2094940763]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:37:58.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:57 smithi028 ceph-mon[24680]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:37:58.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:57 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:58.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:57 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:37:59.215 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:58 smithi192 ceph-mon[26904]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:37:59.215 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:58 smithi192 ceph-mon[26904]: osdmap e30: 5 total, 4 up, 5 in 2024-02-10T05:37:59.215 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:58 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:37:59.215 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:58 smithi192 ceph-mon[26904]: from='osd.4 [v2:172.21.15.192:6800/2094940763,v1:172.21.15.192:6801/2094940763]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:37:59.215 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:58 smithi192 ceph-mon[26904]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:37:59.215 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:58 smithi192 ceph-mon[26904]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:37:59.215 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:37:58 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[31064]: 2024-02-10T05:37:58.904+0000 7f3126a4f700 -1 osd.4 0 waiting for initial osdmap 2024-02-10T05:37:59.215 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:37:58 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[31064]: 2024-02-10T05:37:58.909+0000 7f311f3e2700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:37:59.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:58 smithi028 ceph-mon[24680]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:37:59.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:58 smithi028 ceph-mon[24680]: osdmap e30: 5 total, 4 up, 5 in 2024-02-10T05:37:59.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:58 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:37:59.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:58 smithi028 ceph-mon[24680]: from='osd.4 [v2:172.21.15.192:6800/2094940763,v1:172.21.15.192:6801/2094940763]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:37:59.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:58 smithi028 ceph-mon[24680]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:37:59.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:58 smithi028 ceph-mon[24680]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:38:00.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:59 smithi028 ceph-mon[24680]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-02-10T05:38:00.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:59 smithi028 ceph-mon[24680]: osdmap e31: 5 total, 4 up, 5 in 2024-02-10T05:38:00.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:59 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:38:00.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:37:59 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:38:00.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:59 smithi192 ceph-mon[26904]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-02-10T05:38:00.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:59 smithi192 ceph-mon[26904]: osdmap e31: 5 total, 4 up, 5 in 2024-02-10T05:38:00.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:59 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:38:00.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:37:59 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:38:01.177 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:00 smithi192 ceph-mon[26904]: purged_snaps scrub starts 2024-02-10T05:38:01.178 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:00 smithi192 ceph-mon[26904]: purged_snaps scrub ok 2024-02-10T05:38:01.178 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:00 smithi192 ceph-mon[26904]: osd.4 [v2:172.21.15.192:6800/2094940763,v1:172.21.15.192:6801/2094940763] boot 2024-02-10T05:38:01.178 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:00 smithi192 ceph-mon[26904]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T05:38:01.178 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:00 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:38:01.178 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:00 smithi192 ceph-mon[26904]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:38:01.289 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T05:38:01.289 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T05:38:01.289 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-02-10T05:38:01.289 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-02-10T05:38:01.289 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013541 s, 774 MB/s 2024-02-10T05:38:01.289 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-02-10T05:38:01.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:00 smithi028 ceph-mon[24680]: purged_snaps scrub starts 2024-02-10T05:38:01.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:00 smithi028 ceph-mon[24680]: purged_snaps scrub ok 2024-02-10T05:38:01.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:00 smithi028 ceph-mon[24680]: osd.4 [v2:172.21.15.192:6800/2094940763,v1:172.21.15.192:6801/2094940763] boot 2024-02-10T05:38:01.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:00 smithi028 ceph-mon[24680]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T05:38:01.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:00 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:38:01.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:00 smithi028 ceph-mon[24680]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:38:01.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2024-02-10T05:38:02.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:01 smithi192 ceph-mon[26904]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T05:38:02.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:01 smithi192 ceph-mon[26904]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T05:38:02.210 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:01 smithi028 ceph-mon[24680]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T05:38:02.210 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:01 smithi028 ceph-mon[24680]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T05:38:03.777 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:03 smithi192 ceph-mon[26904]: Detected new or changed devices on smithi192 2024-02-10T05:38:03.777 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:03 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:03.777 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:03 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:38:03.777 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:03 smithi192 ceph-mon[26904]: Adjusting osd_memory_target on smithi192 to 15936M 2024-02-10T05:38:03.777 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:03 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:03.778 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:03 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:03.778 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:03 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:38:03.778 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:03 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:03.778 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:03 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:38:03.778 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:03 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:03.778 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:03 smithi192 ceph-mon[26904]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:38:03.778 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:03 smithi192 ceph-mon[26904]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T05:38:04.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:03 smithi028 ceph-mon[24680]: Detected new or changed devices on smithi192 2024-02-10T05:38:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:03 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:03 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:38:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:03 smithi028 ceph-mon[24680]: Adjusting osd_memory_target on smithi192 to 15936M 2024-02-10T05:38:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:03 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:03 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:03 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:38:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:03 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:03 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:38:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:03 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:03 smithi028 ceph-mon[24680]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:38:04.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:03 smithi028 ceph-mon[24680]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T05:38:05.125 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:04 smithi192 ceph-mon[26904]: osdmap e36: 5 total, 5 up, 5 in 2024-02-10T05:38:05.125 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:04 smithi192 ceph-mon[26904]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:38:05.125 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:38:05.125 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:38:05.125 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:05.125 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:04 smithi192 ceph-mon[26904]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:38:05.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:04 smithi028 ceph-mon[24680]: osdmap e36: 5 total, 5 up, 5 in 2024-02-10T05:38:05.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:04 smithi028 ceph-mon[24680]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:38:05.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:38:05.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:38:05.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:05.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:04 smithi028 ceph-mon[24680]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:38:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:06 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/1923992970' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22dfa292-1c90-428a-8a7e-8bb7fb78735b"}]: dispatch 2024-02-10T05:38:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:06 smithi028 ceph-mon[24680]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22dfa292-1c90-428a-8a7e-8bb7fb78735b"}]: dispatch 2024-02-10T05:38:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:06 smithi028 ceph-mon[24680]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22dfa292-1c90-428a-8a7e-8bb7fb78735b"}]': finished 2024-02-10T05:38:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:06 smithi028 ceph-mon[24680]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T05:38:07.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:38:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:06 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/1923992970' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22dfa292-1c90-428a-8a7e-8bb7fb78735b"}]: dispatch 2024-02-10T05:38:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:06 smithi192 ceph-mon[26904]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22dfa292-1c90-428a-8a7e-8bb7fb78735b"}]: dispatch 2024-02-10T05:38:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:06 smithi192 ceph-mon[26904]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22dfa292-1c90-428a-8a7e-8bb7fb78735b"}]': finished 2024-02-10T05:38:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:06 smithi192 ceph-mon[26904]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T05:38:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:38:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:07 smithi028 ceph-mon[24680]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-02-10T05:38:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:38:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:38:08.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:07 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/2154983151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:38:08.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:07 smithi192 ceph-mon[26904]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-02-10T05:38:08.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:38:08.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:38:08.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:07 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/2154983151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:38:09.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:08 smithi028 ceph-mon[24680]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-10T05:38:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:08 smithi192 ceph-mon[26904]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-10T05:38:11.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:10 smithi192 ceph-mon[26904]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:38:11.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:10 smithi028 ceph-mon[24680]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:38:13.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:12 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:38:13.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:12 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:13.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:12 smithi192 ceph-mon[26904]: Deploying daemon osd.5 on smithi192 2024-02-10T05:38:13.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:12 smithi192 ceph-mon[26904]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:38:13.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:12 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:38:13.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:12 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:13.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:12 smithi028 ceph-mon[24680]: Deploying daemon osd.5 on smithi192 2024-02-10T05:38:13.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:12 smithi028 ceph-mon[24680]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:38:15.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:14 smithi192 ceph-mon[26904]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T05:38:15.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:14 smithi028 ceph-mon[24680]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T05:38:17.190 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:16 smithi192 ceph-mon[26904]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:38:17.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:16 smithi028 ceph-mon[24680]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:38:19.285 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:19.285 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:38:19.285 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:19.285 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:18 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:38:19.285 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:18 smithi192 ceph-mon[26904]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:38:19.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:19.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:38:19.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:19.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:18 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:38:19.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:18 smithi028 ceph-mon[24680]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:38:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:20 smithi192 ceph-mon[26904]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:38:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:20 smithi192 ceph-mon[26904]: from='osd.5 [v2:172.21.15.192:6808/3924027000,v1:172.21.15.192:6809/3924027000]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:38:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:20 smithi192 ceph-mon[26904]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:38:21.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:20 smithi028 ceph-mon[24680]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:38:21.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:20 smithi028 ceph-mon[24680]: from='osd.5 [v2:172.21.15.192:6808/3924027000,v1:172.21.15.192:6809/3924027000]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:38:21.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:20 smithi028 ceph-mon[24680]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:38:21.389 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 5 on host 'smithi192' 2024-02-10T05:38:22.035 DEBUG:teuthology.orchestra.run.smithi192:osd.5> sudo journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.5.service 2024-02-10T05:38:22.038 INFO:tasks.cephadm:Deploying osd.6 on smithi192 with /dev/vg_nvme/lv_2... 2024-02-10T05:38:22.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T05:38:22.689 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:22 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:22.690 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:22 smithi192 ceph-mon[26904]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:38:22.690 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:22 smithi192 ceph-mon[26904]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T05:38:22.690 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:22 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:38:22.690 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:22 smithi192 ceph-mon[26904]: from='osd.5 [v2:172.21.15.192:6808/3924027000,v1:172.21.15.192:6809/3924027000]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:38:22.690 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:22 smithi192 ceph-mon[26904]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:38:22.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:22 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:22.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:22 smithi028 ceph-mon[24680]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:38:22.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:22 smithi028 ceph-mon[24680]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T05:38:22.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:22 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:38:22.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:22 smithi028 ceph-mon[24680]: from='osd.5 [v2:172.21.15.192:6808/3924027000,v1:172.21.15.192:6809/3924027000]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:38:22.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:22 smithi028 ceph-mon[24680]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:38:23.052 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:38:22 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[34048]: 2024-02-10T05:38:22.872+0000 7f441d610700 -1 osd.5 0 waiting for initial osdmap 2024-02-10T05:38:23.052 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:38:22 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[34048]: 2024-02-10T05:38:22.877+0000 7f4415fa3700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:38:23.629 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:23 smithi192 ceph-mon[26904]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:38:23.629 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:23 smithi192 ceph-mon[26904]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-02-10T05:38:23.629 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:23 smithi192 ceph-mon[26904]: osdmap e39: 6 total, 5 up, 6 in 2024-02-10T05:38:23.629 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:23 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:38:23.629 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:23 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:38:23.629 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:23 smithi192 ceph-mon[26904]: osd.5 [v2:172.21.15.192:6808/3924027000,v1:172.21.15.192:6809/3924027000] boot 2024-02-10T05:38:23.629 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:23 smithi192 ceph-mon[26904]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T05:38:23.629 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:23 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:38:23.629 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:23 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:23.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:23 smithi028 ceph-mon[24680]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:38:23.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:23 smithi028 ceph-mon[24680]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-02-10T05:38:23.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:23 smithi028 ceph-mon[24680]: osdmap e39: 6 total, 5 up, 6 in 2024-02-10T05:38:23.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:23 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:38:23.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:23 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:38:23.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:23 smithi028 ceph-mon[24680]: osd.5 [v2:172.21.15.192:6808/3924027000,v1:172.21.15.192:6809/3924027000] boot 2024-02-10T05:38:23.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:23 smithi028 ceph-mon[24680]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T05:38:23.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:23 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:38:23.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:23 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:24.214 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T05:38:24.214 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T05:38:24.214 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-02-10T05:38:24.214 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-02-10T05:38:24.214 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120605 s, 869 MB/s 2024-02-10T05:38:24.214 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-02-10T05:38:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:24 smithi192 ceph-mon[26904]: purged_snaps scrub starts 2024-02-10T05:38:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:24 smithi192 ceph-mon[26904]: purged_snaps scrub ok 2024-02-10T05:38:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:24 smithi192 ceph-mon[26904]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T05:38:24.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:24 smithi028 ceph-mon[24680]: purged_snaps scrub starts 2024-02-10T05:38:24.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:24 smithi028 ceph-mon[24680]: purged_snaps scrub ok 2024-02-10T05:38:24.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:24 smithi028 ceph-mon[24680]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T05:38:25.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:25 smithi192 ceph-mon[26904]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:25.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:25 smithi192 ceph-mon[26904]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T05:38:25.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:25 smithi028 ceph-mon[24680]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:25.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:25 smithi028 ceph-mon[24680]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T05:38:25.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2024-02-10T05:38:26.518 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:26 smithi192 ceph-mon[26904]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:38:26.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:26 smithi028 ceph-mon[24680]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:38:27.781 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:27 smithi028 ceph-mon[24680]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T05:38:27.781 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:27 smithi028 ceph-mon[24680]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:27 smithi192 ceph-mon[26904]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T05:38:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:27 smithi192 ceph-mon[26904]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:28.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:28 smithi028 ceph-mon[24680]: osdmap e44: 6 total, 6 up, 6 in 2024-02-10T05:38:28.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:28 smithi028 ceph-mon[24680]: Detected new or changed devices on smithi192 2024-02-10T05:38:28.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:28.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:38:28.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:38:28.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:28 smithi028 ceph-mon[24680]: Adjusting osd_memory_target on smithi192 to 7968M 2024-02-10T05:38:28.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:28.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:28.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:38:28.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:28.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:38:28.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:28 smithi192 ceph-mon[26904]: osdmap e44: 6 total, 6 up, 6 in 2024-02-10T05:38:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:28 smithi192 ceph-mon[26904]: Detected new or changed devices on smithi192 2024-02-10T05:38:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:38:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:38:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:28 smithi192 ceph-mon[26904]: Adjusting osd_memory_target on smithi192 to 7968M 2024-02-10T05:38:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:38:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:28.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:38:28.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:29.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:29 smithi192 ceph-mon[26904]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:38:29.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:38:29.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:38:29.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:29.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:29 smithi192 ceph-mon[26904]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:29.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:29 smithi028 ceph-mon[24680]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:38:29.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:38:29.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:38:29.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:29.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:29 smithi028 ceph-mon[24680]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:30 smithi192 ceph-mon[26904]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-10T05:38:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:30 smithi192 ceph-mon[26904]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T05:38:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:30 smithi192 ceph-mon[26904]: Cluster is now healthy 2024-02-10T05:38:31.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:30 smithi028 ceph-mon[24680]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-10T05:38:31.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:30 smithi028 ceph-mon[24680]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T05:38:31.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:30 smithi028 ceph-mon[24680]: Cluster is now healthy 2024-02-10T05:38:32.210 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:31 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/3347328231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee381051-900e-4258-8c2c-32c37ef7df99"}]: dispatch 2024-02-10T05:38:32.210 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:31 smithi028 ceph-mon[24680]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee381051-900e-4258-8c2c-32c37ef7df99"}]: dispatch 2024-02-10T05:38:32.210 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:31 smithi028 ceph-mon[24680]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee381051-900e-4258-8c2c-32c37ef7df99"}]': finished 2024-02-10T05:38:32.211 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:31 smithi028 ceph-mon[24680]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T05:38:32.211 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:31 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:38:32.211 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:31 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/2698509142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:38:32.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:31 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/3347328231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee381051-900e-4258-8c2c-32c37ef7df99"}]: dispatch 2024-02-10T05:38:32.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:31 smithi192 ceph-mon[26904]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee381051-900e-4258-8c2c-32c37ef7df99"}]: dispatch 2024-02-10T05:38:32.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:31 smithi192 ceph-mon[26904]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee381051-900e-4258-8c2c-32c37ef7df99"}]': finished 2024-02-10T05:38:32.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:31 smithi192 ceph-mon[26904]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T05:38:32.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:31 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:38:32.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:31 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/2698509142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:38:33.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:32 smithi028 ceph-mon[24680]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-02-10T05:38:33.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:32 smithi192 ceph-mon[26904]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-02-10T05:38:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:34 smithi192 ceph-mon[26904]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:38:35.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:34 smithi028 ceph-mon[24680]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:38:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:36 smithi192 ceph-mon[26904]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-02-10T05:38:37.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:36 smithi028 ceph-mon[24680]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-02-10T05:38:38.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:37 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:38:38.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:37 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:38.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:37 smithi192 ceph-mon[26904]: Deploying daemon osd.6 on smithi192 2024-02-10T05:38:38.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:37 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:38:38.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:37 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:38.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:37 smithi028 ceph-mon[24680]: Deploying daemon osd.6 on smithi192 2024-02-10T05:38:38.988 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:38 smithi192 ceph-mon[26904]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T05:38:39.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:38 smithi028 ceph-mon[24680]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T05:38:41.194 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:40 smithi192 ceph-mon[26904]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:41.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:40 smithi028 ceph-mon[24680]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:38:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:43 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:38:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:43 smithi192 ceph-mon[26904]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:43.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:43.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:38:43.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:43.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:43 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:38:43.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:43 smithi028 ceph-mon[24680]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:45.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:44 smithi192 ceph-mon[26904]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:45.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:44 smithi028 ceph-mon[24680]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:46.273 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:45 smithi192 ceph-mon[26904]: from='osd.6 [v2:172.21.15.192:6816/3956837016,v1:172.21.15.192:6817/3956837016]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:38:46.273 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:45 smithi192 ceph-mon[26904]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:38:46.304 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 6 on host 'smithi192' 2024-02-10T05:38:46.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:45 smithi028 ceph-mon[24680]: from='osd.6 [v2:172.21.15.192:6816/3956837016,v1:172.21.15.192:6817/3956837016]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:38:46.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:45 smithi028 ceph-mon[24680]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:38:46.899 DEBUG:teuthology.orchestra.run.smithi192:osd.6> sudo journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.6.service 2024-02-10T05:38:46.902 INFO:tasks.cephadm:Deploying osd.7 on smithi192 with /dev/vg_nvme/lv_1... 2024-02-10T05:38:46.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T05:38:47.252 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:46 smithi192 ceph-mon[26904]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:38:47.252 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:46 smithi192 ceph-mon[26904]: osdmap e46: 7 total, 6 up, 7 in 2024-02-10T05:38:47.253 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:46 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:38:47.253 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:46 smithi192 ceph-mon[26904]: from='osd.6 [v2:172.21.15.192:6816/3956837016,v1:172.21.15.192:6817/3956837016]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:38:47.253 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:46 smithi192 ceph-mon[26904]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:38:47.253 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:46 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:47.253 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:46 smithi192 ceph-mon[26904]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:47.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:46 smithi028 ceph-mon[24680]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:38:47.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:46 smithi028 ceph-mon[24680]: osdmap e46: 7 total, 6 up, 7 in 2024-02-10T05:38:47.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:46 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:38:47.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:46 smithi028 ceph-mon[24680]: from='osd.6 [v2:172.21.15.192:6816/3956837016,v1:172.21.15.192:6817/3956837016]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:38:47.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:46 smithi028 ceph-mon[24680]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:38:47.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:46 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:47.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:46 smithi028 ceph-mon[24680]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:48.218 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:47 smithi192 ceph-mon[26904]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-02-10T05:38:48.218 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:47 smithi192 ceph-mon[26904]: osdmap e47: 7 total, 6 up, 7 in 2024-02-10T05:38:48.218 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:47 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:38:48.218 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:47 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:38:48.218 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:47 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:48.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:47 smithi028 ceph-mon[24680]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-02-10T05:38:48.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:47 smithi028 ceph-mon[24680]: osdmap e47: 7 total, 6 up, 7 in 2024-02-10T05:38:48.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:47 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:38:48.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:47 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:38:48.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:47 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:49.086 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:48 smithi192 ceph-mon[26904]: purged_snaps scrub starts 2024-02-10T05:38:49.086 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:48 smithi192 ceph-mon[26904]: purged_snaps scrub ok 2024-02-10T05:38:49.086 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:48 smithi192 ceph-mon[26904]: osd.6 [v2:172.21.15.192:6816/3956837016,v1:172.21.15.192:6817/3956837016] boot 2024-02-10T05:38:49.086 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:48 smithi192 ceph-mon[26904]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T05:38:49.086 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:48 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:38:49.086 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:48 smithi192 ceph-mon[26904]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:48 smithi028 ceph-mon[24680]: purged_snaps scrub starts 2024-02-10T05:38:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:48 smithi028 ceph-mon[24680]: purged_snaps scrub ok 2024-02-10T05:38:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:48 smithi028 ceph-mon[24680]: osd.6 [v2:172.21.15.192:6816/3956837016,v1:172.21.15.192:6817/3956837016] boot 2024-02-10T05:38:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:48 smithi028 ceph-mon[24680]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T05:38:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:48 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:38:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:48 smithi028 ceph-mon[24680]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:38:49.555 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T05:38:49.555 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T05:38:49.555 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-02-10T05:38:49.555 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-02-10T05:38:49.555 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122606 s, 855 MB/s 2024-02-10T05:38:49.555 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-02-10T05:38:50.279 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:49 smithi192 ceph-mon[26904]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T05:38:50.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_1 2024-02-10T05:38:50.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:49 smithi028 ceph-mon[24680]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T05:38:51.303 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:50 smithi192 ceph-mon[26904]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T05:38:51.304 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:50 smithi192 ceph-mon[26904]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:38:51.304 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:50 smithi192 ceph-mon[26904]: osdmap e51: 7 total, 7 up, 7 in 2024-02-10T05:38:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:50 smithi028 ceph-mon[24680]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T05:38:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:50 smithi028 ceph-mon[24680]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:38:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:50 smithi028 ceph-mon[24680]: osdmap e51: 7 total, 7 up, 7 in 2024-02-10T05:38:53.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:52 smithi192 ceph-mon[26904]: osdmap e52: 7 total, 7 up, 7 in 2024-02-10T05:38:53.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:52 smithi192 ceph-mon[26904]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:38:53.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:52 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:38:53.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:52 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:38:53.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:52 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:53.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:52 smithi192 ceph-mon[26904]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:38:53.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:52 smithi028 ceph-mon[24680]: osdmap e52: 7 total, 7 up, 7 in 2024-02-10T05:38:53.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:52 smithi028 ceph-mon[24680]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:38:53.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:52 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:38:53.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:52 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:38:53.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:52 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:53.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:52 smithi028 ceph-mon[24680]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:38:54.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: osdmap e53: 7 total, 7 up, 7 in 2024-02-10T05:38:54.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: Detected new or changed devices on smithi192 2024-02-10T05:38:54.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:54.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:38:54.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:38:54.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:38:54.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: Adjusting osd_memory_target on smithi192 to 5312M 2024-02-10T05:38:54.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:54.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:54.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:38:54.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:54.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:38:54.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:53 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: osdmap e53: 7 total, 7 up, 7 in 2024-02-10T05:38:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: Detected new or changed devices on smithi192 2024-02-10T05:38:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:38:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:38:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:38:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: Adjusting osd_memory_target on smithi192 to 5312M 2024-02-10T05:38:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:38:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:38:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:38:54.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:53 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:38:55.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:54 smithi192 ceph-mon[26904]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-02-10T05:38:55.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:54 smithi028 ceph-mon[24680]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-02-10T05:38:56.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:55 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/770470906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "730ed0dc-297a-4794-91d0-f73b616bdb0a"}]: dispatch 2024-02-10T05:38:56.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:55 smithi028 ceph-mon[24680]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "730ed0dc-297a-4794-91d0-f73b616bdb0a"}]: dispatch 2024-02-10T05:38:56.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:55 smithi028 ceph-mon[24680]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "730ed0dc-297a-4794-91d0-f73b616bdb0a"}]': finished 2024-02-10T05:38:56.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:55 smithi028 ceph-mon[24680]: osdmap e54: 8 total, 7 up, 8 in 2024-02-10T05:38:56.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:55 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:38:56.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:55 smithi028 ceph-mon[24680]: from='client.? 172.21.15.192:0/2726511500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:38:56.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:55 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/770470906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "730ed0dc-297a-4794-91d0-f73b616bdb0a"}]: dispatch 2024-02-10T05:38:56.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:55 smithi192 ceph-mon[26904]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "730ed0dc-297a-4794-91d0-f73b616bdb0a"}]: dispatch 2024-02-10T05:38:56.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:55 smithi192 ceph-mon[26904]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "730ed0dc-297a-4794-91d0-f73b616bdb0a"}]': finished 2024-02-10T05:38:56.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:55 smithi192 ceph-mon[26904]: osdmap e54: 8 total, 7 up, 8 in 2024-02-10T05:38:56.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:55 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:38:56.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:55 smithi192 ceph-mon[26904]: from='client.? 172.21.15.192:0/2726511500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:38:57.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:56 smithi028 ceph-mon[24680]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-02-10T05:38:57.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:56 smithi192 ceph-mon[26904]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-02-10T05:38:59.241 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:38:58 smithi192 ceph-mon[26904]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:38:59.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:38:58 smithi028 ceph-mon[24680]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:39:01.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:00 smithi028 ceph-mon[24680]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:39:01.318 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:00 smithi192 ceph-mon[26904]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:39:02.063 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:01 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:39:02.063 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:01 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:39:02.063 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:01 smithi192 ceph-mon[26904]: Deploying daemon osd.7 on smithi192 2024-02-10T05:39:02.210 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:01 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:39:02.210 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:01 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:39:02.210 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:01 smithi028 ceph-mon[24680]: Deploying daemon osd.7 on smithi192 2024-02-10T05:39:03.249 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:02 smithi192 ceph-mon[26904]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T05:39:03.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:02 smithi028 ceph-mon[24680]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-10T05:39:04.979 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:04 smithi192 ceph-mon[26904]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:39:05.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:04 smithi028 ceph-mon[24680]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:39:07.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:07.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:39:07.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:39:07.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:39:07.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:07 smithi028 ceph-mon[24680]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:39:07.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:39:07.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:39:07.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:07.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:39:07.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:39:07.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:39:07.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:07 smithi192 ceph-mon[26904]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:39:07.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:39:07.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:39:09.043 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:08 smithi192 ceph-mon[26904]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:39:09.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:08 smithi028 ceph-mon[24680]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:39:09.809 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 7 on host 'smithi192' 2024-02-10T05:39:10.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:09 smithi192 ceph-mon[26904]: from='osd.7 [v2:172.21.15.192:6824/2256845998,v1:172.21.15.192:6825/2256845998]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:39:10.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:09 smithi192 ceph-mon[26904]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:39:10.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:09 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:10.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:09 smithi028 ceph-mon[24680]: from='osd.7 [v2:172.21.15.192:6824/2256845998,v1:172.21.15.192:6825/2256845998]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:39:10.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:09 smithi028 ceph-mon[24680]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:39:10.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:09 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:10.494 DEBUG:teuthology.orchestra.run.smithi192:osd.7> sudo journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.7.service 2024-02-10T05:39:10.497 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-10T05:39:10.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T05:39:11.084 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:10 smithi028 ceph-mon[24680]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:39:11.084 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:10 smithi028 ceph-mon[24680]: osdmap e55: 8 total, 7 up, 8 in 2024-02-10T05:39:11.084 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:10 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:39:11.085 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:10 smithi028 ceph-mon[24680]: from='osd.7 [v2:172.21.15.192:6824/2256845998,v1:172.21.15.192:6825/2256845998]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:39:11.085 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:10 smithi028 ceph-mon[24680]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:39:11.085 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:10 smithi028 ceph-mon[24680]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:39:11.314 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:10 smithi192 ceph-mon[26904]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:39:11.314 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:10 smithi192 ceph-mon[26904]: osdmap e55: 8 total, 7 up, 8 in 2024-02-10T05:39:11.314 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:10 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:39:11.314 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:10 smithi192 ceph-mon[26904]: from='osd.7 [v2:172.21.15.192:6824/2256845998,v1:172.21.15.192:6825/2256845998]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:39:11.314 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:10 smithi192 ceph-mon[26904]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:39:11.314 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:10 smithi192 ceph-mon[26904]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:39:11.315 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:39:10 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[40120]: 2024-02-10T05:39:10.968+0000 7f27ce19f700 -1 osd.7 0 waiting for initial osdmap 2024-02-10T05:39:11.315 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:39:10 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[40120]: 2024-02-10T05:39:10.975+0000 7f27cab3a700 -1 osd.7 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:39:12.166 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:11 smithi028 ceph-mon[24680]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-02-10T05:39:12.166 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:11 smithi028 ceph-mon[24680]: osdmap e56: 8 total, 7 up, 8 in 2024-02-10T05:39:12.166 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:11 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:39:12.166 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:11 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:39:12.166 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:11 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:11 smithi192 ceph-mon[26904]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-02-10T05:39:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:11 smithi192 ceph-mon[26904]: osdmap e56: 8 total, 7 up, 8 in 2024-02-10T05:39:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:11 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:39:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:11 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:39:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:11 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:12.546 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:39:13.062 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:12 smithi192 ceph-mon[26904]: purged_snaps scrub starts 2024-02-10T05:39:13.062 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:12 smithi192 ceph-mon[26904]: purged_snaps scrub ok 2024-02-10T05:39:13.063 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:12 smithi192 ceph-mon[26904]: osd.7 [v2:172.21.15.192:6824/2256845998,v1:172.21.15.192:6825/2256845998] boot 2024-02-10T05:39:13.063 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:12 smithi192 ceph-mon[26904]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T05:39:13.063 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:12 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:39:13.063 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:12 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2972693372' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:39:13.063 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:12 smithi192 ceph-mon[26904]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:13.063 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:12 smithi192 ceph-mon[26904]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T05:39:13.132 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:12 smithi028 ceph-mon[24680]: purged_snaps scrub starts 2024-02-10T05:39:13.133 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:12 smithi028 ceph-mon[24680]: purged_snaps scrub ok 2024-02-10T05:39:13.133 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:12 smithi028 ceph-mon[24680]: osd.7 [v2:172.21.15.192:6824/2256845998,v1:172.21.15.192:6825/2256845998] boot 2024-02-10T05:39:13.133 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:12 smithi028 ceph-mon[24680]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T05:39:13.133 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:12 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:39:13.133 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:12 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2972693372' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:39:13.133 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:12 smithi028 ceph-mon[24680]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:13.133 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:12 smithi028 ceph-mon[24680]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T05:39:13.134 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1707543551,"num_in_osds":8,"osd_in_since":1707543535,"num_remapped_pgs":1} 2024-02-10T05:39:13.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:39:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:39:15.031 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":59,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","created":"2024-02-10T05:33:13.236671+0000","modified":"2024-02-10T05:39:13.954975+0000","last_up_change":"2024-02-10T05:39:11.961981+0000","last_in_change":"2024-02-10T05:38:55.043178+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:37:15.116032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d2f2678e-aca7-4091-af12-a513153910b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":951842531},{"type":"v1","addr":"172.21.15.28:6803","nonce":951842531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":951842531},{"type":"v1","addr":"172.21.15.28:6805","nonce":951842531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":951842531},{"type":"v1","addr":"172.21.15.28:6809","nonce":951842531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":951842531},{"type":"v1","addr":"172.21.15.28:6807","nonce":951842531}]},"public_addr":"172.21.15.28:6803/951842531","cluster_addr":"172.21.15.28:6805/951842531","heartbeat_back_addr":"172.21.15.28:6809/951842531","heartbeat_front_addr":"172.21.15.28:6807/951842531","state":["exists","up"]},{"osd":1,"uuid":"c619156b-7b30-4239-9ccf-84de21b54db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2828860592},{"type":"v1","addr":"172.21.15.28:6811","nonce":2828860592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2828860592},{"type":"v1","addr":"172.21.15.28:6813","nonce":2828860592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2828860592},{"type":"v1","addr":"172.21.15.28:6817","nonce":2828860592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2828860592},{"type":"v1","addr":"172.21.15.28:6815","nonce":2828860592}]},"public_addr":"172.21.15.28:6811/2828860592","cluster_addr":"172.21.15.28:6813/2828860592","heartbeat_back_addr":"172.21.15.28:6817/2828860592","heartbeat_front_addr":"172.21.15.28:6815/2828860592","state":["exists","up"]},{"osd":2,"uuid":"80df6130-a9d6-4098-a828-3bebb33d2621","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1602733598},{"type":"v1","addr":"172.21.15.28:6819","nonce":1602733598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1602733598},{"type":"v1","addr":"172.21.15.28:6821","nonce":1602733598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1602733598},{"type":"v1","addr":"172.21.15.28:6825","nonce":1602733598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1602733598},{"type":"v1","addr":"172.21.15.28:6823","nonce":1602733598}]},"public_addr":"172.21.15.28:6819/1602733598","cluster_addr":"172.21.15.28:6821/1602733598","heartbeat_back_addr":"172.21.15.28:6825/1602733598","heartbeat_front_addr":"172.21.15.28:6823/1602733598","state":["exists","up"]},{"osd":3,"uuid":"ef46ae3c-2805-410e-ac42-e96f69da62c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":431349139},{"type":"v1","addr":"172.21.15.28:6827","nonce":431349139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":431349139},{"type":"v1","addr":"172.21.15.28:6829","nonce":431349139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":431349139},{"type":"v1","addr":"172.21.15.28:6833","nonce":431349139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":431349139},{"type":"v1","addr":"172.21.15.28:6831","nonce":431349139}]},"public_addr":"172.21.15.28:6827/431349139","cluster_addr":"172.21.15.28:6829/431349139","heartbeat_back_addr":"172.21.15.28:6833/431349139","heartbeat_front_addr":"172.21.15.28:6831/431349139","state":["exists","up"]},{"osd":4,"uuid":"3c916733-e4c8-4ba7-9995-2cdff4590c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2094940763},{"type":"v1","addr":"172.21.15.192:6801","nonce":2094940763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2094940763},{"type":"v1","addr":"172.21.15.192:6803","nonce":2094940763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2094940763},{"type":"v1","addr":"172.21.15.192:6807","nonce":2094940763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2094940763},{"type":"v1","addr":"172.21.15.192:6805","nonce":2094940763}]},"public_addr":"172.21.15.192:6801/2094940763","cluster_addr":"172.21.15.192:6803/2094940763","heartbeat_back_addr":"172.21.15.192:6807/2094940763","heartbeat_front_addr":"172.21.15.192:6805/2094940763","state":["exists","up"]},{"osd":5,"uuid":"22dfa292-1c90-428a-8a7e-8bb7fb78735b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3924027000},{"type":"v1","addr":"172.21.15.192:6809","nonce":3924027000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3924027000},{"type":"v1","addr":"172.21.15.192:6811","nonce":3924027000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3924027000},{"type":"v1","addr":"172.21.15.192:6815","nonce":3924027000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3924027000},{"type":"v1","addr":"172.21.15.192:6813","nonce":3924027000}]},"public_addr":"172.21.15.192:6809/3924027000","cluster_addr":"172.21.15.192:6811/3924027000","heartbeat_back_addr":"172.21.15.192:6815/3924027000","heartbeat_front_addr":"172.21.15.192:6813/3924027000","state":["exists","up"]},{"osd":6,"uuid":"ee381051-900e-4258-8c2c-32c37ef7df99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3956837016},{"type":"v1","addr":"172.21.15.192:6817","nonce":3956837016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3956837016},{"type":"v1","addr":"172.21.15.192:6819","nonce":3956837016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3956837016},{"type":"v1","addr":"172.21.15.192:6823","nonce":3956837016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3956837016},{"type":"v1","addr":"172.21.15.192:6821","nonce":3956837016}]},"public_addr":"172.21.15.192:6817/3956837016","cluster_addr":"172.21.15.192:6819/3956837016","heartbeat_back_addr":"172.21.15.192:6823/3956837016","heartbeat_front_addr":"172.21.15.192:6821/3956837016","state":["exists","up"]},{"osd":7,"uuid":"730ed0dc-297a-4794-91d0-f73b616bdb0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2256845998},{"type":"v1","addr":"172.21.15.192:6825","nonce":2256845998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2256845998},{"type":"v1","addr":"172.21.15.192:6827","nonce":2256845998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2256845998},{"type":"v1","addr":"172.21.15.192:6831","nonce":2256845998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2256845998},{"type":"v1","addr":"172.21.15.192:6829","nonce":2256845998}]},"public_addr":"172.21.15.192:6825/2256845998","cluster_addr":"172.21.15.192:6827/2256845998","heartbeat_back_addr":"172.21.15.192:6831/2256845998","heartbeat_front_addr":"172.21.15.192:6829/2256845998","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:36:12.135209+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:36:43.219152+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:37:10.909480+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:37:36.476489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:37:57.996136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:38:21.852039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:38:46.474179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:39:10.672207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[0,6]}],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3552897338":"2024-02-11T05:35:05.805946+0000","172.21.15.28:6800/3552897338":"2024-02-11T05:35:05.805946+0000","172.21.15.28:0/2537340473":"2024-02-11T05:35:05.805946+0000","172.21.15.28:6801/1164891550":"2024-02-11T05:34:13.374088+0000","172.21.15.28:0/4258486093":"2024-02-11T05:34:13.374088+0000","172.21.15.28:0/2233460746":"2024-02-11T05:34:13.374088+0000","172.21.15.28:6800/1164891550":"2024-02-11T05:34:13.374088+0000","172.21.15.28:0/1159462777":"2024-02-11T05:34:13.374088+0000","172.21.15.28:0/2125122403":"2024-02-11T05:33:43.452051+0000","172.21.15.28:6800/2992116073":"2024-02-11T05:33:43.452051+0000","172.21.15.28:0/3956970017":"2024-02-11T05:35:05.805946+0000","172.21.15.28:6801/2992116073":"2024-02-11T05:33:43.452051+0000","172.21.15.28:0/3645777006":"2024-02-11T05:33:43.452051+0000","172.21.15.28:0/4190450133":"2024-02-11T05:35:05.805946+0000","172.21.15.28:0/1632617597":"2024-02-11T05:33:43.452051+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:39:15.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T05:39:15.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: Detected new or changed devices on smithi192 2024-02-10T05:39:15.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:15.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:39:15.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:39:15.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:39:15.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:39:15.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: Adjusting osd_memory_target on smithi192 to 3984M 2024-02-10T05:39:15.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:15.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:15.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:39:15.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:39:15.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:39:15.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:15.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:14 smithi028 ceph-mon[24680]: pgmap v171: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-02-10T05:39:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T05:39:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: Detected new or changed devices on smithi192 2024-02-10T05:39:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:39:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:39:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:39:15.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:39:15.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: Adjusting osd_memory_target on smithi192 to 3984M 2024-02-10T05:39:15.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:15.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:15.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:39:15.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:39:15.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:39:15.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:39:15.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:14 smithi192 ceph-mon[26904]: pgmap v171: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-02-10T05:39:15.689 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T05:37:15.116032+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-10T05:39:15.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-10T05:39:15.908 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:16.242 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:15 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1148367531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:39:16.243 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:15 smithi028 ceph-mon[24680]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-02-10T05:39:16.243 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:15 smithi028 ceph-mon[24680]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T05:39:16.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:15 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/1148367531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:39:16.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:15 smithi192 ceph-mon[26904]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-02-10T05:39:16.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:15 smithi192 ceph-mon[26904]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T05:39:17.717 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-10T05:39:17.717 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:17 smithi028 ceph-mon[24680]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T05:39:17.717 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:17 smithi028 ceph-mon[24680]: pgmap v174: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-02-10T05:39:17.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:17 smithi192 ceph-mon[26904]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T05:39:17.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:17 smithi192 ceph-mon[26904]: pgmap v174: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-02-10T05:39:18.428 INFO:tasks.cephadm:Setting up client nodes... 2024-02-10T05:39:18.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T05:39:18.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:18 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/864886233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:39:18.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:18 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/864886233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:39:19.743 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:19 smithi028 ceph-mon[24680]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-02-10T05:39:19.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:19 smithi192 ceph-mon[26904]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-02-10T05:39:20.519 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-10T05:39:20.519 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAIDMdlUtnLHhAAOwrRNXN0KjE5PtsONpWMJA== 2024-02-10T05:39:20.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:20 smithi028 ceph-mon[24680]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-02-10T05:39:20.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:20 smithi028 ceph-mon[24680]: Cluster is now healthy 2024-02-10T05:39:20.821 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:20 smithi192 ceph-mon[26904]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-02-10T05:39:20.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:20 smithi192 ceph-mon[26904]: Cluster is now healthy 2024-02-10T05:39:21.124 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-10T05:39:21.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-10T05:39:21.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-10T05:39:21.166 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-10T05:39:21.167 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-10T05:39:21.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-10T05:39:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:21.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:21 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1193152195' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:39:21.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:21 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1193152195' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:39:21.739 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:21 smithi028 ceph-mon[24680]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:39:21.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:21 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/1193152195' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:39:21.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:21 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/1193152195' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:39:21.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:21 smithi192 ceph-mon[26904]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:39:23.160 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:22 smithi028 ceph-mon[24680]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:39:23.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:22 smithi192 ceph-mon[26904]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T05:39:23.422 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:39:24.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:23 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1960711732' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T05:39:24.086 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"active_gid":14202,"active_name":"smithi028.ucuqfx","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3210284990},{"type":"v1","addr":"172.21.15.28:6801","nonce":3210284990}]},"active_addr":"172.21.15.28:6801/3210284990","active_change":"2024-02-10T05:35:05.806297+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14222,"name":"smithi192.hppzcm","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/","prometheus":"http://172.21.15.28:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3773642651}]},{"addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3307142372}]},{"addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3553395726}]},{"addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1250449846}]}]}} 2024-02-10T05:39:24.091 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-10T05:39:24.091 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T05:39:24.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:39:24.308 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:24.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:23 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/1960711732' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T05:39:25.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:24 smithi028 ceph-mon[24680]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-02-10T05:39:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:24 smithi192 ceph-mon[26904]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-02-10T05:39:26.044 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:39:26.045 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":61,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","created":"2024-02-10T05:33:13.236671+0000","modified":"2024-02-10T05:39:16.470999+0000","last_up_change":"2024-02-10T05:39:11.961981+0000","last_in_change":"2024-02-10T05:38:55.043178+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:37:15.116032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d2f2678e-aca7-4091-af12-a513153910b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":951842531},{"type":"v1","addr":"172.21.15.28:6803","nonce":951842531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":951842531},{"type":"v1","addr":"172.21.15.28:6805","nonce":951842531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":951842531},{"type":"v1","addr":"172.21.15.28:6809","nonce":951842531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":951842531},{"type":"v1","addr":"172.21.15.28:6807","nonce":951842531}]},"public_addr":"172.21.15.28:6803/951842531","cluster_addr":"172.21.15.28:6805/951842531","heartbeat_back_addr":"172.21.15.28:6809/951842531","heartbeat_front_addr":"172.21.15.28:6807/951842531","state":["exists","up"]},{"osd":1,"uuid":"c619156b-7b30-4239-9ccf-84de21b54db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2828860592},{"type":"v1","addr":"172.21.15.28:6811","nonce":2828860592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2828860592},{"type":"v1","addr":"172.21.15.28:6813","nonce":2828860592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2828860592},{"type":"v1","addr":"172.21.15.28:6817","nonce":2828860592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2828860592},{"type":"v1","addr":"172.21.15.28:6815","nonce":2828860592}]},"public_addr":"172.21.15.28:6811/2828860592","cluster_addr":"172.21.15.28:6813/2828860592","heartbeat_back_addr":"172.21.15.28:6817/2828860592","heartbeat_front_addr":"172.21.15.28:6815/2828860592","state":["exists","up"]},{"osd":2,"uuid":"80df6130-a9d6-4098-a828-3bebb33d2621","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1602733598},{"type":"v1","addr":"172.21.15.28:6819","nonce":1602733598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1602733598},{"type":"v1","addr":"172.21.15.28:6821","nonce":1602733598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1602733598},{"type":"v1","addr":"172.21.15.28:6825","nonce":1602733598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1602733598},{"type":"v1","addr":"172.21.15.28:6823","nonce":1602733598}]},"public_addr":"172.21.15.28:6819/1602733598","cluster_addr":"172.21.15.28:6821/1602733598","heartbeat_back_addr":"172.21.15.28:6825/1602733598","heartbeat_front_addr":"172.21.15.28:6823/1602733598","state":["exists","up"]},{"osd":3,"uuid":"ef46ae3c-2805-410e-ac42-e96f69da62c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":431349139},{"type":"v1","addr":"172.21.15.28:6827","nonce":431349139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":431349139},{"type":"v1","addr":"172.21.15.28:6829","nonce":431349139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":431349139},{"type":"v1","addr":"172.21.15.28:6833","nonce":431349139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":431349139},{"type":"v1","addr":"172.21.15.28:6831","nonce":431349139}]},"public_addr":"172.21.15.28:6827/431349139","cluster_addr":"172.21.15.28:6829/431349139","heartbeat_back_addr":"172.21.15.28:6833/431349139","heartbeat_front_addr":"172.21.15.28:6831/431349139","state":["exists","up"]},{"osd":4,"uuid":"3c916733-e4c8-4ba7-9995-2cdff4590c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2094940763},{"type":"v1","addr":"172.21.15.192:6801","nonce":2094940763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2094940763},{"type":"v1","addr":"172.21.15.192:6803","nonce":2094940763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2094940763},{"type":"v1","addr":"172.21.15.192:6807","nonce":2094940763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2094940763},{"type":"v1","addr":"172.21.15.192:6805","nonce":2094940763}]},"public_addr":"172.21.15.192:6801/2094940763","cluster_addr":"172.21.15.192:6803/2094940763","heartbeat_back_addr":"172.21.15.192:6807/2094940763","heartbeat_front_addr":"172.21.15.192:6805/2094940763","state":["exists","up"]},{"osd":5,"uuid":"22dfa292-1c90-428a-8a7e-8bb7fb78735b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3924027000},{"type":"v1","addr":"172.21.15.192:6809","nonce":3924027000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3924027000},{"type":"v1","addr":"172.21.15.192:6811","nonce":3924027000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3924027000},{"type":"v1","addr":"172.21.15.192:6815","nonce":3924027000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3924027000},{"type":"v1","addr":"172.21.15.192:6813","nonce":3924027000}]},"public_addr":"172.21.15.192:6809/3924027000","cluster_addr":"172.21.15.192:6811/3924027000","heartbeat_back_addr":"172.21.15.192:6815/3924027000","heartbeat_front_addr":"172.21.15.192:6813/3924027000","state":["exists","up"]},{"osd":6,"uuid":"ee381051-900e-4258-8c2c-32c37ef7df99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3956837016},{"type":"v1","addr":"172.21.15.192:6817","nonce":3956837016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3956837016},{"type":"v1","addr":"172.21.15.192:6819","nonce":3956837016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3956837016},{"type":"v1","addr":"172.21.15.192:6823","nonce":3956837016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3956837016},{"type":"v1","addr":"172.21.15.192:6821","nonce":3956837016}]},"public_addr":"172.21.15.192:6817/3956837016","cluster_addr":"172.21.15.192:6819/3956837016","heartbeat_back_addr":"172.21.15.192:6823/3956837016","heartbeat_front_addr":"172.21.15.192:6821/3956837016","state":["exists","up"]},{"osd":7,"uuid":"730ed0dc-297a-4794-91d0-f73b616bdb0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2256845998},{"type":"v1","addr":"172.21.15.192:6825","nonce":2256845998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2256845998},{"type":"v1","addr":"172.21.15.192:6827","nonce":2256845998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2256845998},{"type":"v1","addr":"172.21.15.192:6831","nonce":2256845998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2256845998},{"type":"v1","addr":"172.21.15.192:6829","nonce":2256845998}]},"public_addr":"172.21.15.192:6825/2256845998","cluster_addr":"172.21.15.192:6827/2256845998","heartbeat_back_addr":"172.21.15.192:6831/2256845998","heartbeat_front_addr":"172.21.15.192:6829/2256845998","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:36:12.135209+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:36:43.219152+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:37:10.909480+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:37:36.476489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:37:57.996136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:38:21.852039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:38:46.474179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:39:10.672207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3552897338":"2024-02-11T05:35:05.805946+0000","172.21.15.28:6800/3552897338":"2024-02-11T05:35:05.805946+0000","172.21.15.28:0/2537340473":"2024-02-11T05:35:05.805946+0000","172.21.15.28:6801/1164891550":"2024-02-11T05:34:13.374088+0000","172.21.15.28:0/4258486093":"2024-02-11T05:34:13.374088+0000","172.21.15.28:0/2233460746":"2024-02-11T05:34:13.374088+0000","172.21.15.28:6800/1164891550":"2024-02-11T05:34:13.374088+0000","172.21.15.28:0/1159462777":"2024-02-11T05:34:13.374088+0000","172.21.15.28:0/2125122403":"2024-02-11T05:33:43.452051+0000","172.21.15.28:6800/2992116073":"2024-02-11T05:33:43.452051+0000","172.21.15.28:0/3956970017":"2024-02-11T05:35:05.805946+0000","172.21.15.28:6801/2992116073":"2024-02-11T05:33:43.452051+0000","172.21.15.28:0/3645777006":"2024-02-11T05:33:43.452051+0000","172.21.15.28:0/4190450133":"2024-02-11T05:35:05.805946+0000","172.21.15.28:0/1632617597":"2024-02-11T05:33:43.452051+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:39:26.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:26 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/456036361' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:39:26.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:26 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/456036361' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:39:26.773 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T05:39:26.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:39:26.999 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:27.351 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:27 smithi028 ceph-mon[24680]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-02-10T05:39:27.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:27 smithi192 ceph-mon[26904]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-02-10T05:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":61,"fsid":"cf1ef3cc-c7d5-11ee-95b8-87774f69a715","created":"2024-02-10T05:33:13.236671+0000","modified":"2024-02-10T05:39:16.470999+0000","last_up_change":"2024-02-10T05:39:11.961981+0000","last_in_change":"2024-02-10T05:38:55.043178+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:37:15.116032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d2f2678e-aca7-4091-af12-a513153910b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":951842531},{"type":"v1","addr":"172.21.15.28:6803","nonce":951842531}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":951842531},{"type":"v1","addr":"172.21.15.28:6805","nonce":951842531}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":951842531},{"type":"v1","addr":"172.21.15.28:6809","nonce":951842531}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":951842531},{"type":"v1","addr":"172.21.15.28:6807","nonce":951842531}]},"public_addr":"172.21.15.28:6803/951842531","cluster_addr":"172.21.15.28:6805/951842531","heartbeat_back_addr":"172.21.15.28:6809/951842531","heartbeat_front_addr":"172.21.15.28:6807/951842531","state":["exists","up"]},{"osd":1,"uuid":"c619156b-7b30-4239-9ccf-84de21b54db3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2828860592},{"type":"v1","addr":"172.21.15.28:6811","nonce":2828860592}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2828860592},{"type":"v1","addr":"172.21.15.28:6813","nonce":2828860592}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2828860592},{"type":"v1","addr":"172.21.15.28:6817","nonce":2828860592}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2828860592},{"type":"v1","addr":"172.21.15.28:6815","nonce":2828860592}]},"public_addr":"172.21.15.28:6811/2828860592","cluster_addr":"172.21.15.28:6813/2828860592","heartbeat_back_addr":"172.21.15.28:6817/2828860592","heartbeat_front_addr":"172.21.15.28:6815/2828860592","state":["exists","up"]},{"osd":2,"uuid":"80df6130-a9d6-4098-a828-3bebb33d2621","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1602733598},{"type":"v1","addr":"172.21.15.28:6819","nonce":1602733598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1602733598},{"type":"v1","addr":"172.21.15.28:6821","nonce":1602733598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1602733598},{"type":"v1","addr":"172.21.15.28:6825","nonce":1602733598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1602733598},{"type":"v1","addr":"172.21.15.28:6823","nonce":1602733598}]},"public_addr":"172.21.15.28:6819/1602733598","cluster_addr":"172.21.15.28:6821/1602733598","heartbeat_back_addr":"172.21.15.28:6825/1602733598","heartbeat_front_addr":"172.21.15.28:6823/1602733598","state":["exists","up"]},{"osd":3,"uuid":"ef46ae3c-2805-410e-ac42-e96f69da62c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":431349139},{"type":"v1","addr":"172.21.15.28:6827","nonce":431349139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":431349139},{"type":"v1","addr":"172.21.15.28:6829","nonce":431349139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":431349139},{"type":"v1","addr":"172.21.15.28:6833","nonce":431349139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":431349139},{"type":"v1","addr":"172.21.15.28:6831","nonce":431349139}]},"public_addr":"172.21.15.28:6827/431349139","cluster_addr":"172.21.15.28:6829/431349139","heartbeat_back_addr":"172.21.15.28:6833/431349139","heartbeat_front_addr":"172.21.15.28:6831/431349139","state":["exists","up"]},{"osd":4,"uuid":"3c916733-e4c8-4ba7-9995-2cdff4590c93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2094940763},{"type":"v1","addr":"172.21.15.192:6801","nonce":2094940763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2094940763},{"type":"v1","addr":"172.21.15.192:6803","nonce":2094940763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2094940763},{"type":"v1","addr":"172.21.15.192:6807","nonce":2094940763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2094940763},{"type":"v1","addr":"172.21.15.192:6805","nonce":2094940763}]},"public_addr":"172.21.15.192:6801/2094940763","cluster_addr":"172.21.15.192:6803/2094940763","heartbeat_back_addr":"172.21.15.192:6807/2094940763","heartbeat_front_addr":"172.21.15.192:6805/2094940763","state":["exists","up"]},{"osd":5,"uuid":"22dfa292-1c90-428a-8a7e-8bb7fb78735b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3924027000},{"type":"v1","addr":"172.21.15.192:6809","nonce":3924027000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3924027000},{"type":"v1","addr":"172.21.15.192:6811","nonce":3924027000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3924027000},{"type":"v1","addr":"172.21.15.192:6815","nonce":3924027000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3924027000},{"type":"v1","addr":"172.21.15.192:6813","nonce":3924027000}]},"public_addr":"172.21.15.192:6809/3924027000","cluster_addr":"172.21.15.192:6811/3924027000","heartbeat_back_addr":"172.21.15.192:6815/3924027000","heartbeat_front_addr":"172.21.15.192:6813/3924027000","state":["exists","up"]},{"osd":6,"uuid":"ee381051-900e-4258-8c2c-32c37ef7df99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3956837016},{"type":"v1","addr":"172.21.15.192:6817","nonce":3956837016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3956837016},{"type":"v1","addr":"172.21.15.192:6819","nonce":3956837016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3956837016},{"type":"v1","addr":"172.21.15.192:6823","nonce":3956837016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3956837016},{"type":"v1","addr":"172.21.15.192:6821","nonce":3956837016}]},"public_addr":"172.21.15.192:6817/3956837016","cluster_addr":"172.21.15.192:6819/3956837016","heartbeat_back_addr":"172.21.15.192:6823/3956837016","heartbeat_front_addr":"172.21.15.192:6821/3956837016","state":["exists","up"]},{"osd":7,"uuid":"730ed0dc-297a-4794-91d0-f73b616bdb0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":2256845998},{"type":"v1","addr":"172.21.15.192:6825","nonce":2256845998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2256845998},{"type":"v1","addr":"172.21.15.192:6827","nonce":2256845998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2256845998},{"type":"v1","addr":"172.21.15.192:6831","nonce":2256845998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2256845998},{"type":"v1","addr":"172.21.15.192:6829","nonce":2256845998}]},"public_addr":"172.21.15.192:6825/2256845998","cluster_addr":"172.21.15.192:6827/2256845998","heartbeat_back_addr":"172.21.15.192:6831/2256845998","heartbeat_front_addr":"172.21.15.192:6829/2256845998","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:36:12.135209+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:36:43.219152+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:37:10.909480+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:37:36.476489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:37:57.996136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:38:21.852039+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:38:46.474179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:39:10.672207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3552897338":"2024-02-11T05:35:05.805946+0000","172.21.15.28:6800/3552897338":"2024-02-11T05:35:05.805946+0000","172.21.15.28:0/2537340473":"2024-02-11T05:35:05.805946+0000","172.21.15.28:6801/1164891550":"2024-02-11T05:34:13.374088+0000","172.21.15.28:0/4258486093":"2024-02-11T05:34:13.374088+0000","172.21.15.28:0/2233460746":"2024-02-11T05:34:13.374088+0000","172.21.15.28:6800/1164891550":"2024-02-11T05:34:13.374088+0000","172.21.15.28:0/1159462777":"2024-02-11T05:34:13.374088+0000","172.21.15.28:0/2125122403":"2024-02-11T05:33:43.452051+0000","172.21.15.28:6800/2992116073":"2024-02-11T05:33:43.452051+0000","172.21.15.28:0/3956970017":"2024-02-11T05:35:05.805946+0000","172.21.15.28:6801/2992116073":"2024-02-11T05:33:43.452051+0000","172.21.15.28:0/3645777006":"2024-02-11T05:33:43.452051+0000","172.21.15.28:0/4190450133":"2024-02-11T05:35:05.805946+0000","172.21.15.28:0/1632617597":"2024-02-11T05:33:43.452051+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:39:29.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:28 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3229330105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:39:29.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:28 smithi028 ceph-mon[24680]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-10T05:39:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:28 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/3229330105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:39:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:28 smithi192 ceph-mon[26904]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-10T05:39:29.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T05:39:29.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T05:39:29.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T05:39:29.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T05:39:29.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T05:39:29.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T05:39:29.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-10T05:39:29.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-10T05:39:29.908 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:29.909 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:29.913 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:29.913 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:29.919 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:29.928 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:30 smithi192 ceph-mon[26904]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:31.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:30 smithi028 ceph-mon[24680]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:33.195 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:32 smithi028 ceph-mon[24680]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:32 smithi192 ceph-mon[26904]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:35.292 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:34 smithi028 ceph-mon[24680]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:34 smithi192 ceph-mon[26904]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:37.305 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:36 smithi028 ceph-mon[24680]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:36 smithi192 ceph-mon[26904]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:37.801 INFO:teuthology.orchestra.run.smithi028.stdout:81604378656 2024-02-10T05:39:37.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T05:39:39.051 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:38 smithi028 ceph-mon[24680]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:39.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:38 smithi192 ceph-mon[26904]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:39.975 INFO:teuthology.orchestra.run.smithi028.stdout:38654705708 2024-02-10T05:39:39.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T05:39:40.844 INFO:teuthology.orchestra.run.smithi028.stdout:244813135880 2024-02-10T05:39:40.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-10T05:39:40.955 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:40 smithi028 ceph-mon[24680]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:41.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:40 smithi192 ceph-mon[26904]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:42.494 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:42.494 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:43.047 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:42 smithi028 ceph-mon[24680]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:43.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:42 smithi192 ceph-mon[26904]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:44.631 INFO:teuthology.orchestra.run.smithi028.stdout:60129542183 2024-02-10T05:39:44.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T05:39:45.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:44 smithi028 ceph-mon[24680]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:45.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:44 smithi192 ceph-mon[26904]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:45.380 INFO:teuthology.orchestra.run.smithi028.stdout:171798691858 2024-02-10T05:39:45.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T05:39:46.754 INFO:teuthology.orchestra.run.smithi028.stdout:115964117020 2024-02-10T05:39:46.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T05:39:47.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:46 smithi028 ceph-mon[24680]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:46 smithi192 ceph-mon[26904]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:47.564 INFO:teuthology.orchestra.run.smithi028.stdout:137438953495 2024-02-10T05:39:47.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T05:39:47.875 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:49.244 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:48 smithi028 ceph-mon[24680]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:48 smithi192 ceph-mon[26904]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:49.455 INFO:teuthology.orchestra.run.smithi028.stdout:38654705710 2024-02-10T05:39:49.486 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:49.486 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:49.487 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:50.040 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:49 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2001574893' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T05:39:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:50.067 INFO:teuthology.orchestra.run.smithi028.stdout:206158430223 2024-02-10T05:39:50.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-10T05:39:50.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:49 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2001574893' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T05:39:51.282 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:50 smithi028 ceph-mon[24680]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:51.311 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705708 got 38654705710 for osd.0 2024-02-10T05:39:51.311 DEBUG:teuthology.parallel:result is None 2024-02-10T05:39:51.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:50 smithi192 ceph-mon[26904]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:53.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:52 smithi028 ceph-mon[24680]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:53.311 INFO:teuthology.orchestra.run.smithi028.stdout:81604378659 2024-02-10T05:39:53.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:52 smithi192 ceph-mon[26904]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:54.240 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:53 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3727537564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T05:39:54.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:53 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/3727537564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T05:39:55.170 INFO:teuthology.orchestra.run.smithi028.stdout:244813135883 2024-02-10T05:39:55.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:54 smithi028 ceph-mon[24680]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:55.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:54 smithi192 ceph-mon[26904]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:56.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:55 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1008689119' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T05:39:56.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:55 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/1008689119' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T05:39:56.724 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:39:56.747 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378656 got 81604378659 for osd.2 2024-02-10T05:39:56.748 DEBUG:teuthology.parallel:result is None 2024-02-10T05:39:57.268 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:56 smithi028 ceph-mon[24680]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:57.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:56 smithi192 ceph-mon[26904]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:57.560 INFO:teuthology.orchestra.run.smithi028.stdout:60129542186 2024-02-10T05:39:57.648 INFO:teuthology.orchestra.run.smithi028.stdout:171798691861 2024-02-10T05:39:58.271 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:57 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2434241235' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T05:39:58.271 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:57 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1861740922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T05:39:58.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:57 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2434241235' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T05:39:58.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:57 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/1861740922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T05:39:59.243 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:58 smithi028 ceph-mon[24680]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:59.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:58 smithi192 ceph-mon[26904]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:39:59.874 INFO:teuthology.orchestra.run.smithi028.stdout:115964117023 2024-02-10T05:40:00.253 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:39:59 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/4006684772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T05:40:00.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:39:59 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/4006684772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T05:40:01.056 INFO:teuthology.orchestra.run.smithi028.stdout:137438953498 2024-02-10T05:40:01.231 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:00 smithi028 ceph-mon[24680]: overall HEALTH_OK 2024-02-10T05:40:01.231 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:00 smithi028 ceph-mon[24680]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:01.260 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691858 got 171798691861 for osd.5 2024-02-10T05:40:01.260 DEBUG:teuthology.parallel:result is None 2024-02-10T05:40:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:00 smithi192 ceph-mon[26904]: overall HEALTH_OK 2024-02-10T05:40:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:00 smithi192 ceph-mon[26904]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:01.860 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953495 got 137438953498 for osd.4 2024-02-10T05:40:01.860 DEBUG:teuthology.parallel:result is None 2024-02-10T05:40:02.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:01 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/661677716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T05:40:02.266 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542183 got 60129542186 for osd.1 2024-02-10T05:40:02.266 DEBUG:teuthology.parallel:result is None 2024-02-10T05:40:02.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:01 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/661677716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T05:40:02.416 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135880 got 244813135883 for osd.7 2024-02-10T05:40:02.416 DEBUG:teuthology.parallel:result is None 2024-02-10T05:40:03.144 INFO:teuthology.orchestra.run.smithi028.stdout:206158430226 2024-02-10T05:40:03.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:02 smithi028 ceph-mon[24680]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:03.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:02 smithi192 ceph-mon[26904]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:03.348 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117020 got 115964117023 for osd.3 2024-02-10T05:40:03.348 DEBUG:teuthology.parallel:result is None 2024-02-10T05:40:04.036 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430223 got 206158430226 for osd.6 2024-02-10T05:40:04.036 DEBUG:teuthology.parallel:result is None 2024-02-10T05:40:04.037 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T05:40:04.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T05:40:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:40:04.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:03 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/4102560802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T05:40:04.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:03 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/4102560802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T05:40:05.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:04 smithi028 ceph-mon[24680]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:05.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:04 smithi192 ceph-mon[26904]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:40:06.029 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-10T05:40:06.652 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-02-10T05:40:04.832857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":49048,"kb_used_data":6488,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":749912168,"statfs":{"total":767960285184,"available":767910060032,"internally_reserved":0,"allocated":6643712,"data_stored":3614200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002333"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":223,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-02-10T05:39:16.479870+0000","last_change":"2024-02-10T05:39:16.479870+0000","last_active":"2024-02-10T05:39:16.479870+0000","last_peered":"2024-02-10T05:39:16.479870+0000","last_clean":"2024-02-10T05:39:16.479870+0000","last_became_active":"2024-02-10T05:39:16.479421+0000","last_became_peered":"2024-02-10T05:39:16.479421+0000","last_unstale":"2024-02-10T05:39:16.479870+0000","last_undegraded":"2024-02-10T05:39:16.479870+0000","last_fullsized":"2024-02-10T05:39:16.479870+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:37:15.166182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:37:15.166182+0000","last_clean_scrub_stamp":"2024-02-10T05:37:15.166182+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:29:22.154504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":57,"seq":244813135885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6072,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739080,"statfs":{"total":95995035648,"available":95988817920,"internally_reserved":0,"allocated":1163264,"data_stored":782355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]}]},{"osd":6,"up_from":48,"seq":206158430227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6200,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738952,"statfs":{"total":95995035648,"available":95988686848,"internally_reserved":0,"allocated":1163264,"data_stored":782355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64500000000000002}]},{"osd":1,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.70399999999999996}]},{"osd":2,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.68799999999999994}]},{"osd":3,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.75800000000000001}]},{"osd":4,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.55800000000000005}]},{"osd":5,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6384,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":630784,"data_stored":253427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:39:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.35599999999999998,"15min":0.33600000000000002},"min":{"1min":0.249,"5min":0.221,"15min":0.221},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.37,"15min":0.33800000000000002},"min":{"1min":0.25900000000000001,"5min":0.223,"15min":0.223},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.39500000000000002}]},{"osd":2,"last update":"Sat Feb 10 05:39:15 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.379,"15min":0.36899999999999999},"min":{"1min":0.30099999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.38,"15min":0.37},"min":{"1min":0.28699999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.379}]},{"osd":3,"last update":"Sat Feb 10 05:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.41999999999999998,"15min":0.41699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.505}]},{"osd":4,"last update":"Sat Feb 10 05:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.876},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.79100000000000004}]},{"osd":5,"last update":"Sat Feb 10 05:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.86299999999999999}]},{"osd":6,"last update":"Sat Feb 10 05:39:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.754}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6904,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738248,"statfs":{"total":95995035648,"available":95987965952,"internally_reserved":0,"allocated":1163264,"data_stored":782355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34000000000000002,"15min":0.33300000000000002},"min":{"1min":0.26900000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.45100000000000001,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.375,"5min":0.36299999999999999,"15min":0.34799999999999998},"min":{"1min":0.255,"5min":0.245,"15min":0.245},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.313}]},{"osd":2,"last update":"Sat Feb 10 05:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40300000000000002,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.27500000000000002}]},{"osd":3,"last update":"Sat Feb 10 05:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42399999999999999},"min":{"1min":0.33200000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.39800000000000002,"15min":0.39700000000000002},"min":{"1min":0.309,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.28699999999999998}]},{"osd":4,"last update":"Sat Feb 10 05:39:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63700000000000001}]},{"osd":5,"last update":"Sat Feb 10 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.503},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.59799999999999998}]},{"osd":6,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.52400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6256,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738896,"statfs":{"total":95995035648,"available":95988629504,"internally_reserved":0,"allocated":630784,"data_stored":253427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:39:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38,"15min":0.375},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.38400000000000001,"15min":0.38300000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.36199999999999999}]},{"osd":1,"last update":"Sat Feb 10 05:39:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.40799999999999997},"min":{"1min":0.311,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.39300000000000002,"15min":0.38500000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.40799999999999997}]},{"osd":3,"last update":"Sat Feb 10 05:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.45400000000000001,"15min":0.45800000000000002},"min":{"1min":0.32700000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.60999999999999999,"5min":0.623,"15min":0.623},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.44700000000000001,"15min":0.45900000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56399999999999995,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.45100000000000001}]},{"osd":4,"last update":"Sat Feb 10 05:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.495},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51100000000000001}]},{"osd":5,"last update":"Sat Feb 10 05:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.52600000000000002}]},{"osd":6,"last update":"Sat Feb 10 05:39:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.63200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5744,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739408,"statfs":{"total":95995035648,"available":95989153792,"internally_reserved":0,"allocated":630784,"data_stored":253427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42699999999999999,"15min":0.42399999999999999},"min":{"1min":0.26300000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42499999999999999,"15min":0.42699999999999999},"min":{"1min":0.27200000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.308}]},{"osd":1,"last update":"Sat Feb 10 05:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.44800000000000001,"15min":0.44600000000000001},"min":{"1min":0.315,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.378},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.441,"15min":0.439},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.44700000000000001}]},{"osd":2,"last update":"Sat Feb 10 05:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33000000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.46800000000000003,"15min":0.46500000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.36399999999999999}]},{"osd":4,"last update":"Sat Feb 10 05:39:02 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.50800000000000001}]},{"osd":5,"last update":"Sat Feb 10 05:39:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":6.1200000000000001,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.55800000000000005}]},{"osd":6,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.73699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953499,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5808,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739344,"statfs":{"total":95995035648,"available":95989088256,"internally_reserved":0,"allocated":630784,"data_stored":253427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:39:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.52800000000000002}]},{"osd":1,"last update":"Sat Feb 10 05:39:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45100000000000001}]},{"osd":2,"last update":"Sat Feb 10 05:39:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.69799999999999995}]},{"osd":3,"last update":"Sat Feb 10 05:39:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Feb 10 05:39:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.67600000000000005}]},{"osd":6,"last update":"Sat Feb 10 05:39:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.499},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.60699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]}]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5680,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739472,"statfs":{"total":95995035648,"available":95989219328,"internally_reserved":0,"allocated":630784,"data_stored":253427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.50800000000000001}]},{"osd":1,"last update":"Sat Feb 10 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Feb 10 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.65700000000000003}]},{"osd":3,"last update":"Sat Feb 10 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.621},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.76600000000000001}]},{"osd":4,"last update":"Sat Feb 10 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.749}]},{"osd":6,"last update":"Sat Feb 10 05:39:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T05:40:06.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T05:40:06.877 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:40:07.214 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:06 smithi028 ceph-mon[24680]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:40:07.214 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:06 smithi028 ceph-mon[24680]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:07.214 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:40:07.214 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:40:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:06 smithi192 ceph-mon[26904]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:40:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:06 smithi192 ceph-mon[26904]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:40:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:40:08.749 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:40:08.751 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-10T05:40:09.124 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:08 smithi028 ceph-mon[24680]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:40:09.124 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:08 smithi028 ceph-mon[24680]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:08 smithi192 ceph-mon[26904]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:40:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:08 smithi192 ceph-mon[26904]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:09.415 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-02-10T05:40:06.833210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":49048,"kb_used_data":6488,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":749912168,"statfs":{"total":767960285184,"available":767910060032,"internally_reserved":0,"allocated":6643712,"data_stored":3614200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002228"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":223,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-02-10T05:39:16.479870+0000","last_change":"2024-02-10T05:39:16.479870+0000","last_active":"2024-02-10T05:39:16.479870+0000","last_peered":"2024-02-10T05:39:16.479870+0000","last_clean":"2024-02-10T05:39:16.479870+0000","last_became_active":"2024-02-10T05:39:16.479421+0000","last_became_peered":"2024-02-10T05:39:16.479421+0000","last_unstale":"2024-02-10T05:39:16.479870+0000","last_undegraded":"2024-02-10T05:39:16.479870+0000","last_fullsized":"2024-02-10T05:39:16.479870+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:37:15.166182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:37:15.166182+0000","last_clean_scrub_stamp":"2024-02-10T05:37:15.166182+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:29:22.154504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":57,"seq":244813135885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6072,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739080,"statfs":{"total":95995035648,"available":95988817920,"internally_reserved":0,"allocated":1163264,"data_stored":782355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]}]},{"osd":6,"up_from":48,"seq":206158430227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6200,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738952,"statfs":{"total":95995035648,"available":95988686848,"internally_reserved":0,"allocated":1163264,"data_stored":782355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64500000000000002}]},{"osd":1,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.70399999999999996}]},{"osd":2,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.68799999999999994}]},{"osd":3,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.75800000000000001}]},{"osd":4,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.55800000000000005}]},{"osd":5,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6384,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":630784,"data_stored":253427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:39:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.35599999999999998,"15min":0.33600000000000002},"min":{"1min":0.249,"5min":0.221,"15min":0.221},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.37,"15min":0.33800000000000002},"min":{"1min":0.25900000000000001,"5min":0.223,"15min":0.223},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Sat Feb 10 05:39:15 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.379,"15min":0.36899999999999999},"min":{"1min":0.30099999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.439,"5min":0.38,"15min":0.37},"min":{"1min":0.28699999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.77800000000000002}]},{"osd":3,"last update":"Sat Feb 10 05:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.41999999999999998,"15min":0.41699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.501},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.47599999999999998,"15min":0.47399999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.53700000000000003}]},{"osd":4,"last update":"Sat Feb 10 05:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.63700000000000001,"15min":0.63},"min":{"1min":0.48699999999999999,"5min":0.441,"15min":0.441},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.63100000000000001,"15min":0.623},"min":{"1min":0.47899999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.749}]},{"osd":5,"last update":"Sat Feb 10 05:39:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Feb 10 05:39:53 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6904,"kb_used_data":1136,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738248,"statfs":{"total":95995035648,"available":95987965952,"internally_reserved":0,"allocated":1163264,"data_stored":782355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34000000000000002,"15min":0.33300000000000002},"min":{"1min":0.26900000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.45100000000000001,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.375,"5min":0.36299999999999999,"15min":0.34799999999999998},"min":{"1min":0.255,"5min":0.245,"15min":0.245},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.313}]},{"osd":2,"last update":"Sat Feb 10 05:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40300000000000002,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.27500000000000002}]},{"osd":3,"last update":"Sat Feb 10 05:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.42699999999999999,"15min":0.42399999999999999},"min":{"1min":0.33200000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.39800000000000002,"15min":0.39700000000000002},"min":{"1min":0.309,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.28699999999999998}]},{"osd":4,"last update":"Sat Feb 10 05:39:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63700000000000001}]},{"osd":5,"last update":"Sat Feb 10 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.503},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.59799999999999998}]},{"osd":6,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.52400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6256,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738896,"statfs":{"total":95995035648,"available":95988629504,"internally_reserved":0,"allocated":630784,"data_stored":253427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:39:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.38,"15min":0.375},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.38400000000000001,"15min":0.38300000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.36199999999999999}]},{"osd":1,"last update":"Sat Feb 10 05:39:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.40999999999999998,"15min":0.40799999999999997},"min":{"1min":0.311,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.39300000000000002,"15min":0.38500000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.40799999999999997}]},{"osd":3,"last update":"Sat Feb 10 05:39:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.45400000000000001,"15min":0.45800000000000002},"min":{"1min":0.32700000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.60999999999999999,"5min":0.623,"15min":0.623},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.44700000000000001,"15min":0.45900000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56399999999999995,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.45100000000000001}]},{"osd":4,"last update":"Sat Feb 10 05:39:05 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.495},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51100000000000001}]},{"osd":5,"last update":"Sat Feb 10 05:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.52600000000000002}]},{"osd":6,"last update":"Sat Feb 10 05:39:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.63200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5744,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739408,"statfs":{"total":95995035648,"available":95989153792,"internally_reserved":0,"allocated":630784,"data_stored":253427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42699999999999999,"15min":0.42399999999999999},"min":{"1min":0.26300000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42499999999999999,"15min":0.42699999999999999},"min":{"1min":0.27200000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.308}]},{"osd":1,"last update":"Sat Feb 10 05:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.44800000000000001,"15min":0.44600000000000001},"min":{"1min":0.315,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.378},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.441,"15min":0.439},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.44700000000000001}]},{"osd":2,"last update":"Sat Feb 10 05:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33000000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.46800000000000003,"15min":0.46500000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.36399999999999999}]},{"osd":4,"last update":"Sat Feb 10 05:39:02 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.50800000000000001}]},{"osd":5,"last update":"Sat Feb 10 05:39:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":6.1200000000000001,"5min":6.1200000000000001,"15min":6.1200000000000001},"last":0.55800000000000005}]},{"osd":6,"last update":"Sat Feb 10 05:39:50 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.73699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953500,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5808,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739344,"statfs":{"total":95995035648,"available":95989088256,"internally_reserved":0,"allocated":630784,"data_stored":253427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.41399999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.67400000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67900000000000005,"5min":0.747,"15min":0.747},"last":0.39100000000000001}]},{"osd":1,"last update":"Sat Feb 10 05:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.41599999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72199999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56599999999999995,"15min":0.57199999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.61499999999999999,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.61099999999999999}]},{"osd":2,"last update":"Sat Feb 10 05:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72199999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.71599999999999997,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54600000000000004}]},{"osd":3,"last update":"Sat Feb 10 05:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.70499999999999996,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58099999999999996,"15min":0.58499999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70499999999999996,"5min":0.874,"15min":0.874},"last":0.57899999999999996}]},{"osd":5,"last update":"Sat Feb 10 05:39:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.59899999999999998}]},{"osd":6,"last update":"Sat Feb 10 05:39:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.67400000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]}]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5680,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739472,"statfs":{"total":95995035648,"available":95989219328,"internally_reserved":0,"allocated":630784,"data_stored":253427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.50800000000000001}]},{"osd":1,"last update":"Sat Feb 10 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Feb 10 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.65700000000000003}]},{"osd":3,"last update":"Sat Feb 10 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.621},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.76600000000000001}]},{"osd":4,"last update":"Sat Feb 10 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.749}]},{"osd":6,"last update":"Sat Feb 10 05:39:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T05:40:09.416 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T05:40:09.416 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-10T05:40:09.416 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-10T05:40:09.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-10T05:40:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/mon.smithi028/config 2024-02-10T05:40:11.005 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:10 smithi028 ceph-mon[24680]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:11.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:10 smithi192 ceph-mon[26904]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:11.473 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:40:11.473 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T05:40:12.059 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-10T05:40:12.060 INFO:tasks.cephadm:Setup complete, yielding 2024-02-10T05:40:12.060 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T05:40:12.072 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T05:40:12.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'ceph orch status' 2024-02-10T05:40:12.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:11 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2821110415' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T05:40:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:11 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2821110415' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T05:40:13.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:12 smithi028 ceph-mon[24680]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:13.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:12 smithi192 ceph-mon[26904]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:13.961 INFO:teuthology.orchestra.run.smithi028.stdout:Backend: cephadm 2024-02-10T05:40:13.961 INFO:teuthology.orchestra.run.smithi028.stdout:Available: Yes 2024-02-10T05:40:13.961 INFO:teuthology.orchestra.run.smithi028.stdout:Paused: No 2024-02-10T05:40:14.551 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:14.551 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:14.552 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:14 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:40:14.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-10T05:40:14.802 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:14.802 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:14.802 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:14 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:40:15.999 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:15 smithi028 ceph-mon[24680]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:15.999 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:15 smithi028 ceph-mon[24680]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:15.999 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:15 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:16.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:15 smithi192 ceph-mon[26904]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:16.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:15 smithi192 ceph-mon[26904]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:16.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:15 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:16.573 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:40:16.573 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (4m) 2m ago 5m 17.5M - ba2b418f427c 110159ccf771 2024-02-10T05:40:16.573 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (5m) 2m ago 5m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:40:16.573 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (4m) 65s ago 4m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (4m) 2m ago 5m 45.2M - 8.3.5 dad864ee21e9 ea3f8ab1d935 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:9283 running (6m) 2m ago 6m 447M - 17.2.0 e1d6a67b021e dc390c86451c 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283 running (4m) 65s ago 4m 413M - 17.2.0 e1d6a67b021e 5de705b3789d 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (6m) 2m ago 7m 42.2M 2048M 17.2.0 e1d6a67b021e a5294cba5afa 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (4m) 65s ago 4m 39.7M 2048M 17.2.0 e1d6a67b021e cf2f3de335d9 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (5m) 2m ago 5m 16.8M - 1dbe0e931976 07432e91f587 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (4m) 65s ago 4m 17.1M - 1dbe0e931976 54c7fecb7980 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (4m) 2m ago 4m 33.3M 3217M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (3m) 2m ago 3m 35.3M 3217M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (3m) 2m ago 3m 30.6M 3217M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (2m) 2m ago 2m 11.0M 3217M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (2m) 65s ago 2m 34.0M 3984M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (119s) 65s ago 118s 32.7M 3984M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (94s) 65s ago 94s 31.1M 3984M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (70s) 65s ago 70s 11.9M 3984M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:40:16.574 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (4m) 2m ago 4m 51.5M - 514e6a882f6e b9d9b936ea2c 2024-02-10T05:40:17.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-10T05:40:18.182 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:17 smithi028 ceph-mon[24680]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:18.182 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:17 smithi028 ceph-mon[24680]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:18.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:17 smithi192 ceph-mon[26904]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:18.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:17 smithi192 ceph-mon[26904]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:19.019 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:18 smithi028 ceph-mon[24680]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-10T05:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-02-10T05:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout:crash 2/2 2m ago 6m * 2024-02-10T05:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-02-10T05:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 2/2 2m ago 6m count:2 2024-02-10T05:40:19.186 INFO:teuthology.orchestra.run.smithi028.stdout:mon 2/2 2m ago 5m smithi028:172.21.15.28=smithi028;smithi192:172.21.15.192=smithi192;count:2 2024-02-10T05:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-02-10T05:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout:osd 8 2m ago - 2024-02-10T05:40:19.187 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-02-10T05:40:19.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:18 smithi192 ceph-mon[26904]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:19.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-10T05:40:20.169 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:19 smithi028 ceph-mon[24680]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:20.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:19 smithi192 ceph-mon[26904]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:21.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:20 smithi028 ceph-mon[24680]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:21.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:20 smithi192 ceph-mon[26904]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:22.049 INFO:teuthology.orchestra.run.smithi028.stdout:HOST ADDR LABELS STATUS 2024-02-10T05:40:22.049 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 172.21.15.28 2024-02-10T05:40:22.049 INFO:teuthology.orchestra.run.smithi028.stdout:smithi192 172.21.15.192 2024-02-10T05:40:22.049 INFO:teuthology.orchestra.run.smithi028.stdout:2 hosts in cluster 2024-02-10T05:40:22.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-10T05:40:23.172 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:22 smithi028 ceph-mon[24680]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:23.172 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:22 smithi028 ceph-mon[24680]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:23.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:22 smithi192 ceph-mon[26904]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:23.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:22 smithi192 ceph-mon[26904]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:24.674 INFO:teuthology.orchestra.run.smithi028.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-02-10T05:40:24.674 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747401F4480DGN 480G LVM detected, locked 2024-02-10T05:40:24.674 INFO:teuthology.orchestra.run.smithi028.stdout:smithi192 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500NN400BGN 400G LVM detected, locked 2024-02-10T05:40:25.006 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:24 smithi028 ceph-mon[24680]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:25.006 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:24 smithi028 ceph-mon[24680]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:24 smithi192 ceph-mon[26904]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:24 smithi192 ceph-mon[26904]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:25.329 INFO:teuthology.run_tasks:Running task vip.exec... 2024-02-10T05:40:25.338 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T05:40:25.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-02-10T05:40:25.368 INFO:teuthology.orchestra.run.smithi028.stderr:+ systemctl stop nfs-server 2024-02-10T05:40:25.376 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi192.front.sepia.ceph.com 2024-02-10T05:40:25.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-02-10T05:40:25.408 INFO:teuthology.orchestra.run.smithi192.stderr:+ systemctl stop nfs-server 2024-02-10T05:40:25.419 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T05:40:25.427 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T05:40:25.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'ceph fs volume create foofs' 2024-02-10T05:40:26.953 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:26 smithi028 ceph-mon[24680]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:26 smithi192 ceph-mon[26904]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:28.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:27 smithi028 ceph-mon[24680]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:28.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:27 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-02-10T05:40:28.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:27 smithi192 ceph-mon[26904]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:28.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:27 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-02-10T05:40:29.278 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-02-10T05:40:29.278 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:28 smithi028 ceph-mon[24680]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T05:40:29.279 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-02-10T05:40:29.279 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:28 smithi028 ceph-mon[24680]: pgmap v211: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:29.279 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:28 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028[24676]: 2024-02-10T05:40:28.931+0000 7f723a9b5700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T05:40:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-02-10T05:40:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:28 smithi192 ceph-mon[26904]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T05:40:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-02-10T05:40:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:28 smithi192 ceph-mon[26904]: pgmap v211: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:29.590 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-02-10T05:40:29.599 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-02-10T05:40:29.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch ls -f json 2024-02-10T05:40:30.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-02-10T05:40:30.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T05:40:30.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-02-10T05:40:30.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T05:40:30.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-10T05:40:30.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T05:40:30.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-02-10T05:40:30.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: fsmap foofs:0 2024-02-10T05:40:30.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: Saving service mds.foofs spec with placement count:2 2024-02-10T05:40:30.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:30.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:30.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:30.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:40:30.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:30.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:30.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi028.xhtowe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T05:40:30.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi028.xhtowe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T05:40:30.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:30.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:29 smithi028 ceph-mon[24680]: Deploying daemon mds.foofs.smithi028.xhtowe on smithi028 2024-02-10T05:40:30.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-02-10T05:40:30.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T05:40:30.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-02-10T05:40:30.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T05:40:30.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-10T05:40:30.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T05:40:30.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-02-10T05:40:30.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: fsmap foofs:0 2024-02-10T05:40:30.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: Saving service mds.foofs spec with placement count:2 2024-02-10T05:40:30.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:30.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:30.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:30.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:40:30.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:30.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:30.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi028.xhtowe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T05:40:30.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi028.xhtowe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T05:40:30.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:30.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:29 smithi192 ceph-mon[26904]: Deploying daemon mds.foofs.smithi028.xhtowe on smithi028 2024-02-10T05:40:31.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:30 smithi028 ceph-mon[24680]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T05:40:31.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:30 smithi028 ceph-mon[24680]: pgmap v215: 65 pgs: 26 creating+peering, 15 active+clean, 24 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:30 smithi192 ceph-mon[26904]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T05:40:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:30 smithi192 ceph-mon[26904]: pgmap v215: 65 pgs: 26 creating+peering, 15 active+clean, 24 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:32.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:31 smithi028 ceph-mon[24680]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T05:40:32.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:31 smithi192 ceph-mon[26904]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T05:40:32.614 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:40:32.614 INFO:teuthology.orchestra.run.smithi028.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-10T05:34:02.862424Z", "last_refresh": "2024-02-10T05:37:39.168872Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-10T05:33:56.744993Z", "last_refresh": "2024-02-10T05:37:39.168939Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-02-10T05:33:59.400736Z", "last_refresh": "2024-02-10T05:37:39.169003Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-02-10T05:40:28.954248Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-10T05:40:28.946230Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-10T05:33:54.804111Z", "last_refresh": "2024-02-10T05:37:39.168798Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi028:172.21.15.28=smithi028", "smithi192:172.21.15.192=smithi192"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-10T05:34:45.473096Z", "last_refresh": "2024-02-10T05:37:39.168655Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-10T05:34:01.255398Z", "last_refresh": "2024-02-10T05:37:39.169086Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-02-10T05:37:39.169216Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-10T05:33:57.959700Z", "last_refresh": "2024-02-10T05:37:39.169152Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-10T05:40:33.262 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:32 smithi028 ceph-mon[24680]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:40:33.262 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:32 smithi028 ceph-mon[24680]: pgmap v217: 65 pgs: 26 creating+peering, 32 active+clean, 7 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:33.286 INFO:tasks.cephadm:mds.foofs has 0/2 2024-02-10T05:40:33.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:32 smithi192 ceph-mon[26904]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:40:33.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:32 smithi192 ceph-mon[26904]: pgmap v217: 65 pgs: 26 creating+peering, 32 active+clean, 7 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:34.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch ls -f json 2024-02-10T05:40:34.857 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:34.857 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi192.hdcbzw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T05:40:34.857 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi192.hdcbzw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T05:40:34.857 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:34.857 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: Deploying daemon mds.foofs.smithi192.hdcbzw on smithi192 2024-02-10T05:40:34.857 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: mds.? [v2:172.21.15.28:6834/350665837,v1:172.21.15.28:6835/350665837] up:boot 2024-02-10T05:40:34.857 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: daemon mds.foofs.smithi028.xhtowe assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-02-10T05:40:34.857 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T05:40:34.857 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-10T05:40:34.857 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: Cluster is now healthy 2024-02-10T05:40:34.857 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: fsmap foofs:0 1 up:standby 2024-02-10T05:40:34.857 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi028.xhtowe"}]: dispatch 2024-02-10T05:40:34.858 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:creating} 2024-02-10T05:40:34.858 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:34 smithi028 ceph-mon[24680]: daemon mds.foofs.smithi028.xhtowe is now active in filesystem foofs as rank 0 2024-02-10T05:40:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi192.hdcbzw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T05:40:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi192.hdcbzw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-10T05:40:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: Deploying daemon mds.foofs.smithi192.hdcbzw on smithi192 2024-02-10T05:40:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: mds.? [v2:172.21.15.28:6834/350665837,v1:172.21.15.28:6835/350665837] up:boot 2024-02-10T05:40:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: daemon mds.foofs.smithi028.xhtowe assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-02-10T05:40:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T05:40:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-10T05:40:35.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: Cluster is now healthy 2024-02-10T05:40:35.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: fsmap foofs:0 1 up:standby 2024-02-10T05:40:35.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi028.xhtowe"}]: dispatch 2024-02-10T05:40:35.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:creating} 2024-02-10T05:40:35.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:34 smithi192 ceph-mon[26904]: daemon mds.foofs.smithi028.xhtowe is now active in filesystem foofs as rank 0 2024-02-10T05:40:35.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:35 smithi192 ceph-mon[26904]: pgmap v218: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:35.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:35 smithi192 ceph-mon[26904]: mds.? [v2:172.21.15.28:6834/350665837,v1:172.21.15.28:6835/350665837] up:active 2024-02-10T05:40:35.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:35 smithi192 ceph-mon[26904]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 2024-02-10T05:40:35.881 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:35 smithi028 ceph-mon[24680]: pgmap v218: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:35.881 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:35 smithi028 ceph-mon[24680]: mds.? [v2:172.21.15.28:6834/350665837,v1:172.21.15.28:6835/350665837] up:active 2024-02-10T05:40:35.881 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:35 smithi028 ceph-mon[24680]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 2024-02-10T05:40:36.450 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:40:36.450 INFO:teuthology.orchestra.run.smithi028.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-10T05:34:02.862424Z", "last_refresh": "2024-02-10T05:37:39.168872Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-10T05:33:56.744993Z", "last_refresh": "2024-02-10T05:37:39.168939Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-02-10T05:33:59.400736Z", "last_refresh": "2024-02-10T05:37:39.169003Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-02-10T05:40:28.954248Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-10T05:40:28.946230Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-10T05:33:54.804111Z", "last_refresh": "2024-02-10T05:37:39.168798Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi028:172.21.15.28=smithi028", "smithi192:172.21.15.192=smithi192"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-10T05:34:45.473096Z", "last_refresh": "2024-02-10T05:37:39.168655Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-10T05:34:01.255398Z", "last_refresh": "2024-02-10T05:37:39.169086Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-02-10T05:37:39.169216Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-10T05:33:57.959700Z", "last_refresh": "2024-02-10T05:37:39.169152Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-10T05:40:37.101 INFO:tasks.cephadm:mds.foofs has 0/2 2024-02-10T05:40:37.390 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:37 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:37.390 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:37 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:37.390 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:37 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:37.390 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:37 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:40:37.390 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:37 smithi028 ceph-mon[24680]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:40:37.390 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:37 smithi028 ceph-mon[24680]: mds.? [v2:172.21.15.192:6832/853070905,v1:172.21.15.192:6833/853070905] up:boot 2024-02-10T05:40:37.391 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:37 smithi028 ceph-mon[24680]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 1 up:standby 2024-02-10T05:40:37.391 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:37 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi192.hdcbzw"}]: dispatch 2024-02-10T05:40:37.391 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:37 smithi028 ceph-mon[24680]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 1 up:standby 2024-02-10T05:40:37.391 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:37 smithi028 ceph-mon[24680]: pgmap v219: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 388 B/s wr, 0 op/s 2024-02-10T05:40:37.391 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:37 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:37.507 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:37 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:37.507 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:37 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:37.507 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:37 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:37.507 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:37 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:40:37.507 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:37 smithi192 ceph-mon[26904]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:40:37.507 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:37 smithi192 ceph-mon[26904]: mds.? [v2:172.21.15.192:6832/853070905,v1:172.21.15.192:6833/853070905] up:boot 2024-02-10T05:40:37.507 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:37 smithi192 ceph-mon[26904]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 1 up:standby 2024-02-10T05:40:37.507 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:37 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi192.hdcbzw"}]: dispatch 2024-02-10T05:40:37.507 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:37 smithi192 ceph-mon[26904]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 1 up:standby 2024-02-10T05:40:37.507 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:37 smithi192 ceph-mon[26904]: pgmap v219: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 388 B/s wr, 0 op/s 2024-02-10T05:40:37.507 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:37 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:38.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch ls -f json 2024-02-10T05:40:39.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:39 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:39.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:39 smithi028 ceph-mon[24680]: pgmap v220: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 345 B/s wr, 1 op/s 2024-02-10T05:40:39.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:39 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:39 smithi192 ceph-mon[26904]: pgmap v220: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 345 B/s wr, 1 op/s 2024-02-10T05:40:40.435 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:40:40.435 INFO:teuthology.orchestra.run.smithi028.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-10T05:34:02.862424Z", "last_refresh": "2024-02-10T05:37:39.168872Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-10T05:33:56.744993Z", "last_refresh": "2024-02-10T05:37:39.168939Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-02-10T05:33:59.400736Z", "last_refresh": "2024-02-10T05:37:39.169003Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-02-10T05:40:28.954248Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-10T05:40:28.946230Z", "last_refresh": "2024-02-10T05:40:38.205573Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-10T05:33:54.804111Z", "last_refresh": "2024-02-10T05:37:39.168798Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi028:172.21.15.28=smithi028", "smithi192:172.21.15.192=smithi192"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-10T05:34:45.473096Z", "last_refresh": "2024-02-10T05:37:39.168655Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-10T05:34:01.255398Z", "last_refresh": "2024-02-10T05:37:39.169086Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-02-10T05:37:39.169216Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-10T05:33:57.959700Z", "last_refresh": "2024-02-10T05:37:39.169152Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-10T05:40:41.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:40 smithi028 ceph-mon[24680]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:40:41.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:40 smithi028 ceph-mon[24680]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-10T05:40:41.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:40 smithi192 ceph-mon[26904]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:40:41.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:40 smithi192 ceph-mon[26904]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-10T05:40:41.494 INFO:tasks.cephadm:mds.foofs has 1/2 2024-02-10T05:40:42.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch ls -f json 2024-02-10T05:40:43.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:42 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:43.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:42 smithi028 ceph-mon[24680]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T05:40:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:42 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:42 smithi192 ceph-mon[26904]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T05:40:45.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:44 smithi028 ceph-mon[24680]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T05:40:45.201 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:40:45.201 INFO:teuthology.orchestra.run.smithi028.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-10T05:34:02.862424Z", "last_refresh": "2024-02-10T05:40:45.136749Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-10T05:33:56.744993Z", "last_refresh": "2024-02-10T05:40:38.204959Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-02-10T05:33:59.400736Z", "last_refresh": "2024-02-10T05:40:45.136966Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-02-10T05:40:28.954248Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-10T05:40:28.946230Z", "last_refresh": "2024-02-10T05:40:38.205573Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-10T05:33:54.804111Z", "last_refresh": "2024-02-10T05:40:38.205119Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi028:172.21.15.28=smithi028", "smithi192:172.21.15.192=smithi192"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-10T05:34:45.473096Z", "last_refresh": "2024-02-10T05:40:38.205195Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-10T05:34:01.255398Z", "last_refresh": "2024-02-10T05:40:38.205261Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-02-10T05:40:38.205325Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-10T05:33:57.959700Z", "last_refresh": "2024-02-10T05:40:45.137156Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-10T05:40:45.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:44 smithi192 ceph-mon[26904]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T05:40:45.809 INFO:tasks.cephadm:mds.foofs has 2/2 2024-02-10T05:40:45.809 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T05:40:45.818 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T05:40:45.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-02-10T05:40:46.456 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:46 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:46.456 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:46 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:46.456 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:46 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:46.456 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:46 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:46.456 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:46 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:40:46.457 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:46 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:46.457 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:46 smithi028 ceph-mon[24680]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:40:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:46 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:46 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:46 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:46 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:46 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:40:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:46 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:46 smithi192 ceph-mon[26904]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:40:47.402 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:47 smithi028 ceph-mon[24680]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T05:40:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:47 smithi192 ceph-mon[26904]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-10T05:40:48.520 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:48 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-02-10T05:40:48.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:48 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-02-10T05:40:49.252 INFO:teuthology.orchestra.run.smithi028.stdout:NFS Cluster Created Successfully 2024-02-10T05:40:49.501 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:49 smithi028 ceph-mon[24680]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:49.501 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:49 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-02-10T05:40:49.501 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:49 smithi028 ceph-mon[24680]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T05:40:49.501 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:49 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-02-10T05:40:49.501 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:49 smithi028 ceph-mon[24680]: pgmap v226: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 3 op/s 2024-02-10T05:40:49.501 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:49 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-02-10T05:40:49.501 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:49 smithi028 ceph-mon[24680]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T05:40:49.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:49 smithi192 ceph-mon[26904]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:49.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:49 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-02-10T05:40:49.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:49 smithi192 ceph-mon[26904]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T05:40:49.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:49 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-02-10T05:40:49.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:49 smithi192 ceph-mon[26904]: pgmap v226: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 3 op/s 2024-02-10T05:40:49.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:49 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-02-10T05:40:49.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:49 smithi192 ceph-mon[26904]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T05:40:49.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-02-10T05:40:50.520 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: Saving service nfs.foo spec with placement count:2 2024-02-10T05:40:50.520 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: Creating key for client.nfs.foo.0.0.smithi028.tzitkz 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi028.tzitkz", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi028.tzitkz", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T05:40:50.521 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T05:40:50.522 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: Rados config object exists: conf-nfs.foo 2024-02-10T05:40:50.522 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: Creating key for client.nfs.foo.0.0.smithi028.tzitkz-rgw 2024-02-10T05:40:50.522 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi028.tzitkz-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:40:50.522 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi028.tzitkz-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:40:50.522 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:50.522 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: Deploying daemon nfs.foo.0.0.smithi028.tzitkz on smithi028 2024-02-10T05:40:50.522 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:50 smithi028 ceph-mon[24680]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T05:40:50.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: Saving service nfs.foo spec with placement count:2 2024-02-10T05:40:50.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:50.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:50.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:50.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:40:50.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:50.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:50.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: Creating key for client.nfs.foo.0.0.smithi028.tzitkz 2024-02-10T05:40:50.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi028.tzitkz", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T05:40:50.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi028.tzitkz", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: Rados config object exists: conf-nfs.foo 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: Creating key for client.nfs.foo.0.0.smithi028.tzitkz-rgw 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi028.tzitkz-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi028.tzitkz-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: Deploying daemon nfs.foo.0.0.smithi028.tzitkz on smithi028 2024-02-10T05:40:50.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:50 smithi192 ceph-mon[26904]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T05:40:51.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:51 smithi028 ceph-mon[24680]: mgrmap e20: smithi028.ucuqfx(active, since 5m), standbys: smithi192.hppzcm 2024-02-10T05:40:51.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:51 smithi028 ceph-mon[24680]: pgmap v229: 97 pgs: 18 creating+peering, 9 unknown, 70 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:51 smithi192 ceph-mon[26904]: mgrmap e20: smithi028.ucuqfx(active, since 5m), standbys: smithi192.hppzcm 2024-02-10T05:40:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:51 smithi192 ceph-mon[26904]: pgmap v229: 97 pgs: 18 creating+peering, 9 unknown, 70 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:53.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:52 smithi028 ceph-mon[24680]: pgmap v230: 97 pgs: 18 creating+peering, 79 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:53.206 INFO:teuthology.orchestra.run.smithi028.stderr:Invalid command: Unexpected argument '--clusterid' 2024-02-10T05:40:53.207 INFO:teuthology.orchestra.run.smithi028.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-02-10T05:40:53.207 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-10T05:40:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:52 smithi192 ceph-mon[26904]: pgmap v230: 97 pgs: 18 creating+peering, 79 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:40:53.560 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:40:53.561 INFO:teuthology.orchestra.run.smithi028.stdout: "bind": "/fake", 2024-02-10T05:40:53.561 INFO:teuthology.orchestra.run.smithi028.stdout: "fs": "foofs", 2024-02-10T05:40:53.561 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-02-10T05:40:53.561 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster": "foo", 2024-02-10T05:40:53.561 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "RW" 2024-02-10T05:40:53.561 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:40:54.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-02-10T05:40:54.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:54.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:54 smithi028 ceph-mon[24680]: Creating key for client.nfs.foo.1.0.smithi192.xgmliu 2024-02-10T05:40:54.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi192.xgmliu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T05:40:54.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi192.xgmliu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-02-10T05:40:54.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:54 smithi028 ceph-mon[24680]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-02-10T05:40:54.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T05:40:54.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T05:40:54.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:54.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:54 smithi028 ceph-mon[24680]: from='client.14522 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:54.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-02-10T05:40:54.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:54 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-02-10T05:40:54.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:54.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:54 smithi192 ceph-mon[26904]: Creating key for client.nfs.foo.1.0.smithi192.xgmliu 2024-02-10T05:40:54.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi192.xgmliu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T05:40:54.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi192.xgmliu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-02-10T05:40:54.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:54 smithi192 ceph-mon[26904]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-02-10T05:40:54.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T05:40:54.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T05:40:54.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:54.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:54 smithi192 ceph-mon[26904]: from='client.14522 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:54.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-02-10T05:40:54.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:54 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-02-10T05:40:55.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:55 smithi028 ceph-mon[24680]: pgmap v231: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T05:40:55.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:55 smithi192 ceph-mon[26904]: pgmap v231: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T05:40:56.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:56 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T05:40:56.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:56 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T05:40:56.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:56 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi192.xgmliu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:40:56.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:56 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi192.xgmliu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:40:56.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:56 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:56.821 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:56 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T05:40:56.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:56 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T05:40:56.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:56 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi192.xgmliu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:40:56.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:56 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi192.xgmliu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:40:56.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:56 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:57.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:57 smithi028 ceph-mon[24680]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:57.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:57 smithi028 ceph-mon[24680]: Rados config object exists: conf-nfs.foo 2024-02-10T05:40:57.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:57 smithi028 ceph-mon[24680]: Creating key for client.nfs.foo.1.0.smithi192.xgmliu-rgw 2024-02-10T05:40:57.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:57 smithi028 ceph-mon[24680]: Deploying daemon nfs.foo.1.0.smithi192.xgmliu on smithi192 2024-02-10T05:40:57.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:57 smithi028 ceph-mon[24680]: pgmap v232: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-02-10T05:40:57.811 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:57 smithi192 ceph-mon[26904]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:57.811 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:57 smithi192 ceph-mon[26904]: Rados config object exists: conf-nfs.foo 2024-02-10T05:40:57.811 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:57 smithi192 ceph-mon[26904]: Creating key for client.nfs.foo.1.0.smithi192.xgmliu-rgw 2024-02-10T05:40:57.811 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:57 smithi192 ceph-mon[26904]: Deploying daemon nfs.foo.1.0.smithi192.xgmliu on smithi192 2024-02-10T05:40:57.811 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:57 smithi192 ceph-mon[26904]: pgmap v232: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-02-10T05:40:58.789 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:58 smithi192 ceph-mon[26904]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:58.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:58 smithi028 ceph-mon[24680]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:59.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:59 smithi028 ceph-mon[24680]: pgmap v233: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-02-10T05:40:59.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:59 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:59.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:59 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:59.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:59 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:59.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:40:59 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:40:59.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:59 smithi192 ceph-mon[26904]: pgmap v233: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-02-10T05:40:59.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:59 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:40:59.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:59 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:59.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:59 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:59.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:40:59 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:00.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:00 smithi028 ceph-mon[24680]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:00.716 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:00 smithi192 ceph-mon[26904]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:01.745 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:01 smithi028 ceph-mon[24680]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:01.746 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:01 smithi028 ceph-mon[24680]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-02-10T05:41:01.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:01 smithi192 ceph-mon[26904]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:01.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:01 smithi192 ceph-mon[26904]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-02-10T05:41:02.583 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:02 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:02.584 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:02 smithi028 ceph-mon[24680]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:02.584 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:02 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:02.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:02 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:02.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:02 smithi192 ceph-mon[26904]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:02.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:02 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:03.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:03 smithi028 ceph-mon[24680]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-02-10T05:41:03.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:03 smithi192 ceph-mon[26904]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-02-10T05:41:04.239 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo ?:2049 2/2 2s ago 14s count:2 2024-02-10T05:41:04.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:04 smithi028 ceph-mon[24680]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:04.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:04.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:04.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:04.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:04.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:04.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:04.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:04.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:04 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:04.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:04 smithi192 ceph-mon[26904]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:04.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:04.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:04.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:04.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:04.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:04.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:04.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:04.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:04 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:04.964 INFO:teuthology.run_tasks:Running task vip.exec... 2024-02-10T05:41:04.974 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T05:41:04.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-02-10T05:41:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:+ mkdir /mnt/foo 2024-02-10T05:41:05.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-02-10T05:41:05.085 INFO:teuthology.orchestra.run.smithi028.stderr:++ hostname 2024-02-10T05:41:05.085 INFO:teuthology.orchestra.run.smithi028.stderr:+ mount -t nfs smithi028:/fake /mnt/foo -o sync 2024-02-10T05:41:05.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-02-10T05:41:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:+ echo test 2024-02-10T05:41:05.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-02-10T05:41:05.645 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:05 smithi028 ceph-mon[24680]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:05.646 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:05 smithi028 ceph-mon[24680]: pgmap v236: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-02-10T05:41:05.648 INFO:teuthology.orchestra.run.smithi028.stderr:+ sync 2024-02-10T05:41:05.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:05 smithi192 ceph-mon[26904]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:05.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:05 smithi192 ceph-mon[26904]: pgmap v236: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-02-10T05:41:05.900 INFO:teuthology.run_tasks:Running task parallel... 2024-02-10T05:41:05.910 INFO:teuthology.task.parallel:starting parallel... 2024-02-10T05:41:05.910 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-10T05:41:05.911 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-10T05:41:05.911 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T05:41:05.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-10T05:41:05.912 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-10T05:41:05.912 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-02-10T05:41:05.913 INFO:teuthology.task.exec:Executing custom commands... 2024-02-10T05:41:05.913 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T05:41:05.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-02-10T05:41:05.984 INFO:teuthology.orchestra.run.smithi028.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-02-10T05:41:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:41:05.984 INFO:teuthology.orchestra.run.smithi028.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-02-10T05:41:05.992 INFO:teuthology.orchestra.run.smithi028.stdout:3 of 5 processes prepared for launch 0 sec 2024-02-10T05:41:06.093 INFO:teuthology.orchestra.run.smithi028.stdout:5 of 5 processes prepared for launch 0 sec 2024-02-10T05:41:06.093 INFO:teuthology.orchestra.run.smithi028.stdout:releasing clients 2024-02-10T05:41:07.000 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-10T05:41:07.001 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T05:41:07.001 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T05:41:07.001 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-10T05:41:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 146 31.29 MB/sec warmup 1 sec latency 52.272 ms 2024-02-10T05:41:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-10T05:41:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-10T05:41:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-10T05:41:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-10T05:41:08.048 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:07 smithi028 ceph-mon[24680]: pgmap v237: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T05:41:08.048 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-02-10T05:41:08.048 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-02-10T05:41:08.048 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-02-10T05:41:08.048 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-02-10T05:41:08.048 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:07 smithi028 ceph-mon[24680]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T05:41:08.048 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:41:08.049 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:41:08.049 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:07 smithi028 ceph-mon[24680]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T05:41:08.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 291 26.83 MB/sec warmup 2 sec latency 30.266 ms 2024-02-10T05:41:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:07 smithi192 ceph-mon[26904]: pgmap v237: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T05:41:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-02-10T05:41:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-02-10T05:41:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-02-10T05:41:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-02-10T05:41:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:07 smithi192 ceph-mon[26904]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T05:41:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:41:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:41:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:07 smithi192 ceph-mon[26904]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T05:41:09.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 428 25.10 MB/sec warmup 3 sec latency 31.639 ms 2024-02-10T05:41:09.315 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:09 smithi028 ceph-mon[24680]: pgmap v240: 97 pgs: 97 active+clean; 5.1 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 578 KiB/s wr, 9 op/s 2024-02-10T05:41:09.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-10T05:41:09.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:09 smithi192 ceph-mon[26904]: pgmap v240: 97 pgs: 97 active+clean; 5.1 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 578 KiB/s wr, 9 op/s 2024-02-10T05:41:10.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 562 24.19 MB/sec warmup 4 sec latency 32.655 ms 2024-02-10T05:41:11.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:10 smithi192 ceph-mon[26904]: pgmap v241: 97 pgs: 97 active+clean; 62 MiB data, 109 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 7.9 MiB/s wr, 338 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:41:11.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 957 21.95 MB/sec warmup 5 sec latency 23.489 ms 2024-02-10T05:41:11.154 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:10 smithi028 ceph-mon[24680]: pgmap v241: 97 pgs: 97 active+clean; 62 MiB data, 109 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 7.9 MiB/s wr, 338 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:41:12.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 1560 21.76 MB/sec warmup 6 sec latency 50.295 ms 2024-02-10T05:41:13.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 2473 20.28 MB/sec warmup 7 sec latency 25.123 ms 2024-02-10T05:41:13.181 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:12 smithi028 ceph-mon[24680]: pgmap v242: 97 pgs: 97 active+clean; 91 MiB data, 133 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 12 MiB/s wr, 444 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:41:13.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:12 smithi192 ceph-mon[26904]: pgmap v242: 97 pgs: 97 active+clean; 91 MiB data, 133 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 12 MiB/s wr, 444 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:41:14.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 3467 23.25 MB/sec warmup 8 sec latency 48.624 ms 2024-02-10T05:41:14.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-10T05:41:14.943 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:14 smithi028 ceph-mon[24680]: pgmap v243: 97 pgs: 97 active+clean; 119 MiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 17 MiB/s wr, 799 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:41:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 3910 22.22 MB/sec warmup 9 sec latency 32.923 ms 2024-02-10T05:41:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:14 smithi192 ceph-mon[26904]: pgmap v243: 97 pgs: 97 active+clean; 119 MiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 17 MiB/s wr, 799 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:41:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 4597 20.76 MB/sec warmup 10 sec latency 23.480 ms 2024-02-10T05:41:16.954 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:16 smithi028 ceph-mon[24680]: pgmap v244: 97 pgs: 97 active+clean; 123 MiB data, 206 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 14 MiB/s wr, 677 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:41:17.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 5473 20.70 MB/sec warmup 11 sec latency 16.291 ms 2024-02-10T05:41:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:16 smithi192 ceph-mon[26904]: pgmap v244: 97 pgs: 97 active+clean; 123 MiB data, 206 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 14 MiB/s wr, 677 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:41:18.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 6110 20.16 MB/sec warmup 12 sec latency 24.900 ms 2024-02-10T05:41:19.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 6927 21.36 MB/sec warmup 13 sec latency 29.154 ms 2024-02-10T05:41:19.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph mgr module enable nfs --force' 2024-02-10T05:41:19.222 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:19 smithi028 ceph-mon[24680]: pgmap v245: 97 pgs: 97 active+clean; 130 MiB data, 236 MiB used, 715 GiB / 715 GiB avail; 840 B/s rd, 14 MiB/s wr, 743 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:41:19.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:19 smithi192 ceph-mon[26904]: pgmap v245: 97 pgs: 97 active+clean; 130 MiB data, 236 MiB used, 715 GiB / 715 GiB avail; 840 B/s rd, 14 MiB/s wr, 743 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:41:20.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 7448 21.07 MB/sec warmup 14 sec latency 31.795 ms 2024-02-10T05:41:20.996 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:20 smithi028 ceph-mon[24680]: pgmap v246: 97 pgs: 97 active+clean; 142 MiB data, 289 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 16 MiB/s wr, 1k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:41:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:20 smithi192 ceph-mon[26904]: pgmap v246: 97 pgs: 97 active+clean; 142 MiB data, 289 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 16 MiB/s wr, 1k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:41:21.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 8014 20.07 MB/sec warmup 15 sec latency 39.309 ms 2024-02-10T05:41:22.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 8609 19.93 MB/sec warmup 16 sec latency 34.731 ms 2024-02-10T05:41:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:module 'nfs' is already enabled 2024-02-10T05:41:22.777 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:22 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/857271478' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-02-10T05:41:23.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:22 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/857271478' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-02-10T05:41:23.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 9321 19.24 MB/sec warmup 17 sec latency 48.621 ms 2024-02-10T05:41:23.881 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:23 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/857271478' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-02-10T05:41:23.882 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:23 smithi028 ceph-mon[24680]: mgrmap e21: smithi028.ucuqfx(active, since 6m), standbys: smithi192.hppzcm 2024-02-10T05:41:23.882 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:23 smithi028 ceph-mon[24680]: pgmap v247: 97 pgs: 97 active+clean; 146 MiB data, 316 MiB used, 715 GiB / 715 GiB avail; 6.6 KiB/s rd, 13 MiB/s wr, 970 op/s 2024-02-10T05:41:23.972 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:23 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/857271478' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-02-10T05:41:23.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:23 smithi192 ceph-mon[26904]: mgrmap e21: smithi028.ucuqfx(active, since 6m), standbys: smithi192.hppzcm 2024-02-10T05:41:23.973 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:23 smithi192 ceph-mon[26904]: pgmap v247: 97 pgs: 97 active+clean; 146 MiB data, 316 MiB used, 715 GiB / 715 GiB avail; 6.6 KiB/s rd, 13 MiB/s wr, 970 op/s 2024-02-10T05:41:23.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-10T05:41:24.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 9967 19.41 MB/sec warmup 18 sec latency 19.330 ms 2024-02-10T05:41:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 10870 20.43 MB/sec warmup 19 sec latency 16.513 ms 2024-02-10T05:41:25.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:24 smithi028 ceph-mon[24680]: pgmap v248: 97 pgs: 97 active+clean; 144 MiB data, 391 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 980 op/s 2024-02-10T05:41:25.266 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:24 smithi192 ceph-mon[26904]: pgmap v248: 97 pgs: 97 active+clean; 144 MiB data, 391 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 980 op/s 2024-02-10T05:41:26.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 11341 19.65 MB/sec warmup 20 sec latency 22.176 ms 2024-02-10T05:41:27.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 12012 19.44 MB/sec warmup 21 sec latency 17.964 ms 2024-02-10T05:41:27.158 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:26 smithi028 ceph-mon[24680]: pgmap v249: 97 pgs: 97 active+clean; 141 MiB data, 418 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 769 op/s 2024-02-10T05:41:27.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:26 smithi192 ceph-mon[26904]: pgmap v249: 97 pgs: 97 active+clean; 141 MiB data, 418 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.5 MiB/s wr, 769 op/s 2024-02-10T05:41:27.926 INFO:teuthology.orchestra.run.smithi028.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:41:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 12821 19.20 MB/sec warmup 22 sec latency 31.588 ms 2024-02-10T05:41:28.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:28 smithi028 ceph-mon[24680]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:28.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:28 smithi028 ceph-mon[24680]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:41:28.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:28.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:28.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:28.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:28.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:28.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:28 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:28.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:28 smithi028 ceph-mon[24680]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:41:28.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:28 smithi028 ceph-mon[24680]: pgmap v250: 97 pgs: 97 active+clean; 148 MiB data, 447 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 880 op/s 2024-02-10T05:41:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 13226 18.90 MB/sec warmup 23 sec latency 41.728 ms 2024-02-10T05:41:29.168 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-10T05:41:29.169 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T05:41:29.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-10T05:41:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:28 smithi192 ceph-mon[26904]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:28 smithi192 ceph-mon[26904]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:41:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:28 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:41:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:28 smithi192 ceph-mon[26904]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:41:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:28 smithi192 ceph-mon[26904]: pgmap v250: 97 pgs: 97 active+clean; 148 MiB data, 447 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 880 op/s 2024-02-10T05:41:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 13793 19.18 MB/sec warmup 24 sec latency 26.762 ms 2024-02-10T05:41:30.963 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:30 smithi028 ceph-mon[24680]: pgmap v251: 97 pgs: 97 active+clean; 155 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-02-10T05:41:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:30 smithi192 ceph-mon[26904]: pgmap v251: 97 pgs: 97 active+clean; 155 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.02k op/s 2024-02-10T05:41:31.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 14325 19.27 MB/sec warmup 25 sec latency 55.786 ms 2024-02-10T05:41:32.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 14689 18.75 MB/sec warmup 26 sec latency 25.884 ms 2024-02-10T05:41:32.769 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:41:32.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:32 smithi028 ceph-mon[24680]: from='client.24369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:32.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:32 smithi028 ceph-mon[24680]: pgmap v252: 97 pgs: 97 active+clean; 159 MiB data, 481 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 829 op/s 2024-02-10T05:41:33.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 15231 18.32 MB/sec warmup 27 sec latency 62.078 ms 2024-02-10T05:41:33.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:32 smithi192 ceph-mon[26904]: from='client.24369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:32 smithi192 ceph-mon[26904]: pgmap v252: 97 pgs: 97 active+clean; 159 MiB data, 481 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 829 op/s 2024-02-10T05:41:33.620 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:41:33.620 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (5m) 29s ago 6m 24.2M - ba2b418f427c 110159ccf771 2024-02-10T05:41:33.620 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (6m) 29s ago 6m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:41:33.620 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (6m) 32s ago 6m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (5m) 29s ago 6m 51.7M - 8.3.5 dad864ee21e9 ea3f8ab1d935 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (61s) 29s ago 59s 14.7M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (57s) 32s ago 57s 12.1M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:9283 running (8m) 29s ago 8m 460M - 17.2.0 e1d6a67b021e dc390c86451c 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283 running (6m) 32s ago 6m 414M - 17.2.0 e1d6a67b021e 5de705b3789d 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (8m) 29s ago 8m 43.5M 2048M 17.2.0 e1d6a67b021e a5294cba5afa 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (6m) 32s ago 6m 37.3M 2048M 17.2.0 e1d6a67b021e cf2f3de335d9 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (41s) 29s ago 40s 53.1M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (35s) 32s ago 34s 48.7M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (6m) 29s ago 6m 18.5M - 1dbe0e931976 07432e91f587 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (5m) 32s ago 5m 17.4M - 1dbe0e931976 54c7fecb7980 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (5m) 29s ago 5m 43.6M 3217M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (4m) 29s ago 4m 43.9M 3217M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:41:33.621 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (4m) 29s ago 4m 39.5M 3217M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:41:33.622 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (4m) 29s ago 4m 40.8M 3217M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:41:33.622 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (3m) 32s ago 3m 41.2M 3984M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:41:33.622 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (3m) 32s ago 3m 38.5M 3984M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:41:33.622 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (2m) 32s ago 2m 39.2M 3984M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:41:33.622 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (2m) 32s ago 2m 41.3M 3984M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:41:33.622 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (5m) 29s ago 5m 62.4M - 514e6a882f6e b9d9b936ea2c 2024-02-10T05:41:34.093 INFO:teuthology.orchestra.run.smithi028.stdout: 5 16060 18.37 MB/sec warmup 28 sec latency 22.866 ms 2024-02-10T05:41:34.231 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:34 smithi028 ceph-mon[24680]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:34.232 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:34 smithi028 ceph-mon[24680]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:34.269 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:41:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:41:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:41:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:41:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:41:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:41:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:41:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:41:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:41:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:41:34.269 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:41:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:41:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:41:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:41:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-02-10T05:41:34.270 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:41:34.270 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:41:34.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:34 smithi192 ceph-mon[26904]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:34.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:34 smithi192 ceph-mon[26904]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:34.813 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:41:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90", 2024-02-10T05:41:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:41:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-02-10T05:41:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "", 2024-02-10T05:41:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 image" 2024-02-10T05:41:34.814 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:41:35.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 16659 18.12 MB/sec warmup 29 sec latency 22.123 ms 2024-02-10T05:41:35.260 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:35 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/4250525033' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:41:35.261 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:35 smithi028 ceph-mon[24680]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:35.261 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:35 smithi028 ceph-mon[24680]: pgmap v253: 97 pgs: 97 active+clean; 172 MiB data, 523 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 842 op/s 2024-02-10T05:41:35.394 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:41:35.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:35 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/4250525033' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:41:35.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:35 smithi192 ceph-mon[26904]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:35.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:35 smithi192 ceph-mon[26904]: pgmap v253: 97 pgs: 97 active+clean; 172 MiB data, 523 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 842 op/s 2024-02-10T05:41:36.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 17114 18.18 MB/sec warmup 30 sec latency 33.276 ms 2024-02-10T05:41:36.274 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:36 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3791041271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:41:36.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:36 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/3791041271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:41:37.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 17701 18.38 MB/sec warmup 31 sec latency 53.948 ms 2024-02-10T05:41:37.286 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:37 smithi028 ceph-mon[24680]: pgmap v254: 97 pgs: 97 active+clean; 169 MiB data, 561 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 8.0 MiB/s wr, 785 op/s 2024-02-10T05:41:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:37 smithi192 ceph-mon[26904]: pgmap v254: 97 pgs: 97 active+clean; 169 MiB data, 561 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 8.0 MiB/s wr, 785 op/s 2024-02-10T05:41:38.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 18098 18.19 MB/sec warmup 32 sec latency 23.173 ms 2024-02-10T05:41:39.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 18477 17.72 MB/sec warmup 33 sec latency 26.100 ms 2024-02-10T05:41:39.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:39 smithi028 ceph-mon[24680]: pgmap v255: 97 pgs: 97 active+clean; 174 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 8.7 MiB/s wr, 879 op/s 2024-02-10T05:41:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:39 smithi192 ceph-mon[26904]: pgmap v255: 97 pgs: 97 active+clean; 174 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 8.8 KiB/s rd, 8.7 MiB/s wr, 879 op/s 2024-02-10T05:41:40.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 19212 17.81 MB/sec warmup 34 sec latency 20.859 ms 2024-02-10T05:41:41.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:40 smithi192 ceph-mon[26904]: pgmap v256: 97 pgs: 97 active+clean; 179 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 895 op/s 2024-02-10T05:41:41.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:40 smithi028 ceph-mon[24680]: pgmap v256: 97 pgs: 97 active+clean; 179 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 895 op/s 2024-02-10T05:41:41.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 20113 17.64 MB/sec warmup 35 sec latency 18.159 ms 2024-02-10T05:41:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 21065 18.18 MB/sec warmup 36 sec latency 22.900 ms 2024-02-10T05:41:42.944 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:42 smithi028 ceph-mon[24680]: pgmap v257: 97 pgs: 97 active+clean; 179 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 7.6 MiB/s wr, 671 op/s 2024-02-10T05:41:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 21713 18.32 MB/sec warmup 37 sec latency 21.196 ms 2024-02-10T05:41:43.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:42 smithi192 ceph-mon[26904]: pgmap v257: 97 pgs: 97 active+clean; 179 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 7.6 MiB/s wr, 671 op/s 2024-02-10T05:41:44.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 22261 17.98 MB/sec warmup 38 sec latency 29.536 ms 2024-02-10T05:41:44.951 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:44 smithi028 ceph-mon[24680]: pgmap v258: 97 pgs: 97 active+clean; 190 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 889 op/s 2024-02-10T05:41:45.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 23105 18.09 MB/sec warmup 39 sec latency 21.667 ms 2024-02-10T05:41:45.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:44 smithi192 ceph-mon[26904]: pgmap v258: 97 pgs: 97 active+clean; 190 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 889 op/s 2024-02-10T05:41:46.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 23882 18.00 MB/sec warmup 40 sec latency 27.265 ms 2024-02-10T05:41:46.965 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:46 smithi028 ceph-mon[24680]: pgmap v259: 97 pgs: 97 active+clean; 189 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 729 op/s 2024-02-10T05:41:47.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 24754 18.42 MB/sec warmup 41 sec latency 13.492 ms 2024-02-10T05:41:47.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:46 smithi192 ceph-mon[26904]: pgmap v259: 97 pgs: 97 active+clean; 189 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 729 op/s 2024-02-10T05:41:48.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 25410 18.45 MB/sec warmup 42 sec latency 30.065 ms 2024-02-10T05:41:49.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 26097 18.31 MB/sec warmup 43 sec latency 13.835 ms 2024-02-10T05:41:49.232 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:49 smithi028 ceph-mon[24680]: pgmap v260: 97 pgs: 97 active+clean; 196 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 9.4 MiB/s wr, 904 op/s 2024-02-10T05:41:49.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:49 smithi192 ceph-mon[26904]: pgmap v260: 97 pgs: 97 active+clean; 196 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 9.4 MiB/s wr, 904 op/s 2024-02-10T05:41:50.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 26827 18.25 MB/sec warmup 44 sec latency 24.110 ms 2024-02-10T05:41:50.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:50 smithi028 ceph-mon[24680]: pgmap v261: 97 pgs: 97 active+clean; 200 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 975 op/s 2024-02-10T05:41:51.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:50 smithi192 ceph-mon[26904]: pgmap v261: 97 pgs: 97 active+clean; 200 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 11 MiB/s wr, 975 op/s 2024-02-10T05:41:51.094 INFO:teuthology.orchestra.run.smithi028.stdout: 5 27438 18.13 MB/sec warmup 45 sec latency 29.147 ms 2024-02-10T05:41:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 28134 18.40 MB/sec warmup 46 sec latency 18.585 ms 2024-02-10T05:41:53.014 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:52 smithi028 ceph-mon[24680]: pgmap v262: 97 pgs: 97 active+clean; 204 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 959 op/s 2024-02-10T05:41:53.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 28794 18.49 MB/sec warmup 47 sec latency 24.196 ms 2024-02-10T05:41:53.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:52 smithi192 ceph-mon[26904]: pgmap v262: 97 pgs: 97 active+clean; 204 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 959 op/s 2024-02-10T05:41:54.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 29370 18.27 MB/sec warmup 48 sec latency 20.642 ms 2024-02-10T05:41:55.028 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:54 smithi028 ceph-mon[24680]: pgmap v263: 97 pgs: 97 active+clean; 207 MiB data, 874 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-02-10T05:41:55.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 29975 18.17 MB/sec warmup 49 sec latency 27.159 ms 2024-02-10T05:41:55.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:54 smithi192 ceph-mon[26904]: pgmap v263: 97 pgs: 97 active+clean; 207 MiB data, 874 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-02-10T05:41:56.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 30654 18.09 MB/sec warmup 50 sec latency 29.054 ms 2024-02-10T05:41:57.051 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:56 smithi028 ceph-mon[24680]: pgmap v264: 97 pgs: 97 active+clean; 210 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 775 op/s 2024-02-10T05:41:57.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 31170 18.01 MB/sec warmup 51 sec latency 31.551 ms 2024-02-10T05:41:57.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:56 smithi192 ceph-mon[26904]: pgmap v264: 97 pgs: 97 active+clean; 210 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 775 op/s 2024-02-10T05:41:58.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 31842 18.21 MB/sec warmup 52 sec latency 20.565 ms 2024-02-10T05:41:59.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 32329 18.20 MB/sec warmup 53 sec latency 28.846 ms 2024-02-10T05:41:59.299 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:41:59 smithi028 ceph-mon[24680]: pgmap v265: 97 pgs: 97 active+clean; 214 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 841 op/s 2024-02-10T05:41:59.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:41:59 smithi192 ceph-mon[26904]: pgmap v265: 97 pgs: 97 active+clean; 214 MiB data, 909 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 841 op/s 2024-02-10T05:42:00.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 32775 17.94 MB/sec warmup 54 sec latency 43.954 ms 2024-02-10T05:42:01.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:00 smithi192 ceph-mon[26904]: pgmap v266: 97 pgs: 97 active+clean; 222 MiB data, 938 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 897 op/s 2024-02-10T05:42:01.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:00 smithi028 ceph-mon[24680]: pgmap v266: 97 pgs: 97 active+clean; 222 MiB data, 938 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 897 op/s 2024-02-10T05:42:01.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 33338 17.84 MB/sec warmup 55 sec latency 22.785 ms 2024-02-10T05:42:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 33943 17.76 MB/sec warmup 56 sec latency 30.274 ms 2024-02-10T05:42:03.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 34477 17.65 MB/sec warmup 57 sec latency 49.377 ms 2024-02-10T05:42:03.116 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:02 smithi028 ceph-mon[24680]: pgmap v267: 97 pgs: 97 active+clean; 226 MiB data, 950 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 785 op/s 2024-02-10T05:42:03.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:02 smithi192 ceph-mon[26904]: pgmap v267: 97 pgs: 97 active+clean; 226 MiB data, 950 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 785 op/s 2024-02-10T05:42:04.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 35146 17.82 MB/sec warmup 58 sec latency 23.924 ms 2024-02-10T05:42:05.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 35775 17.93 MB/sec warmup 59 sec latency 22.760 ms 2024-02-10T05:42:05.180 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:04 smithi028 ceph-mon[24680]: pgmap v268: 97 pgs: 97 active+clean; 230 MiB data, 1015 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.8 MiB/s wr, 809 op/s 2024-02-10T05:42:05.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:04 smithi192 ceph-mon[26904]: pgmap v268: 97 pgs: 97 active+clean; 230 MiB data, 1015 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.8 MiB/s wr, 809 op/s 2024-02-10T05:42:05.802 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:42:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 36028 17.69 MB/sec warmup 60 sec latency 43.795 ms 2024-02-10T05:42:06.193 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:42:06.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:06 smithi028 ceph-mon[24680]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-10T05:42:06.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:06 smithi028 ceph-mon[24680]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T05:42:06.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:06.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:42:06.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:06 smithi028 ceph-mon[24680]: Upgrade: Need to upgrade myself (mgr.smithi028.ucuqfx) 2024-02-10T05:42:06.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:06 smithi028 ceph-mon[24680]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 on smithi192 2024-02-10T05:42:06.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:06 smithi028 ceph-mon[24680]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:06.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:42:06.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:06 smithi192 ceph-mon[26904]: Upgrade: Target is version 19.0.0-1361-g290212a1 (unknown) 2024-02-10T05:42:06.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:06 smithi192 ceph-mon[26904]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, digests ['quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58'] 2024-02-10T05:42:06.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:06.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:42:06.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:06 smithi192 ceph-mon[26904]: Upgrade: Need to upgrade myself (mgr.smithi028.ucuqfx) 2024-02-10T05:42:06.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:06 smithi192 ceph-mon[26904]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 on smithi192 2024-02-10T05:42:06.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:06 smithi192 ceph-mon[26904]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:06.655 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:42:06.655 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (6m) 62s ago 7m 24.2M - ba2b418f427c 110159ccf771 2024-02-10T05:42:06.655 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (7m) 62s ago 7m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:42:06.655 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (6m) 65s ago 6m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:42:06.655 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (5m) 62s ago 7m 51.7M - 8.3.5 dad864ee21e9 ea3f8ab1d935 2024-02-10T05:42:06.655 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (94s) 62s ago 92s 14.7M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:42:06.655 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (90s) 65s ago 90s 12.1M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:9283 running (8m) 62s ago 8m 460M - 17.2.0 e1d6a67b021e dc390c86451c 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283 running (6m) 65s ago 6m 414M - 17.2.0 e1d6a67b021e 5de705b3789d 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (8m) 62s ago 8m 43.5M 2048M 17.2.0 e1d6a67b021e a5294cba5afa 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (6m) 65s ago 6m 37.3M 2048M 17.2.0 e1d6a67b021e cf2f3de335d9 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (74s) 62s ago 73s 53.1M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (68s) 65s ago 67s 48.7M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (7m) 62s ago 7m 18.5M - 1dbe0e931976 07432e91f587 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (6m) 65s ago 6m 17.4M - 1dbe0e931976 54c7fecb7980 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (5m) 62s ago 5m 43.6M 3217M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (5m) 62s ago 5m 43.9M 3217M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (5m) 62s ago 4m 39.5M 3217M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (4m) 62s ago 4m 40.8M 3217M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:42:06.656 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (4m) 65s ago 4m 41.2M 3984M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:42:06.657 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (3m) 65s ago 3m 38.5M 3984M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:42:06.657 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (3m) 65s ago 3m 39.2M 3984M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:42:06.657 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (3m) 65s ago 3m 41.3M 3984M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:42:06.657 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (6m) 62s ago 6m 62.4M - 514e6a882f6e b9d9b936ea2c 2024-02-10T05:42:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 36434 17.51 MB/sec warmup 61 sec latency 35.991 ms 2024-02-10T05:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:42:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-02-10T05:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:42:07.165 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:42:07.208 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:07 smithi028 ceph-mon[24680]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:07.208 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:07 smithi028 ceph-mon[24680]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:07.208 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:07 smithi028 ceph-mon[24680]: pgmap v269: 97 pgs: 97 active+clean; 229 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 677 op/s 2024-02-10T05:42:07.209 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:42:07.209 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:42:07.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:07 smithi192 ceph-mon[26904]: from='client.14636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:07 smithi192 ceph-mon[26904]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:07 smithi192 ceph-mon[26904]: pgmap v269: 97 pgs: 97 active+clean; 229 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 677 op/s 2024-02-10T05:42:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:42:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:42:07.700 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-02-10T05:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "0/23 daemons upgraded", 2024-02-10T05:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 image on host smithi192" 2024-02-10T05:42:07.701 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:42:08.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 36956 17.44 MB/sec warmup 62 sec latency 43.449 ms 2024-02-10T05:42:08.246 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:08 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/872347190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:42:08.246 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:08 smithi028 ceph-mon[24680]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:08.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:08 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/872347190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:42:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:08 smithi192 ceph-mon[26904]: from='client.14648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:08.345 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:42:09.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 37754 17.39 MB/sec warmup 63 sec latency 28.821 ms 2024-02-10T05:42:09.252 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:09 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/119382838' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:42:09.252 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:09 smithi028 ceph-mon[24680]: pgmap v270: 97 pgs: 97 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 713 op/s 2024-02-10T05:42:09.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:09 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/119382838' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:42:09.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:09 smithi192 ceph-mon[26904]: pgmap v270: 97 pgs: 97 active+clean; 230 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 713 op/s 2024-02-10T05:42:10.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 38433 17.47 MB/sec warmup 64 sec latency 25.032 ms 2024-02-10T05:42:11.010 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:10 smithi028 ceph-mon[24680]: pgmap v271: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.2 MiB/s wr, 942 op/s 2024-02-10T05:42:11.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:10 smithi192 ceph-mon[26904]: pgmap v271: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.2 MiB/s wr, 942 op/s 2024-02-10T05:42:11.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 39105 17.60 MB/sec warmup 65 sec latency 29.553 ms 2024-02-10T05:42:12.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 39607 17.55 MB/sec warmup 66 sec latency 22.798 ms 2024-02-10T05:42:13.023 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:12 smithi028 ceph-mon[24680]: pgmap v272: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.9 MiB/s wr, 729 op/s 2024-02-10T05:42:13.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 40225 17.40 MB/sec warmup 67 sec latency 32.406 ms 2024-02-10T05:42:13.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:12 smithi192 ceph-mon[26904]: pgmap v272: 97 pgs: 97 active+clean; 243 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.9 MiB/s wr, 729 op/s 2024-02-10T05:42:14.096 INFO:teuthology.orchestra.run.smithi028.stdout: 5 40805 17.43 MB/sec warmup 68 sec latency 76.054 ms 2024-02-10T05:42:15.087 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:14 smithi028 ceph-mon[24680]: pgmap v273: 97 pgs: 97 active+clean; 250 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.8 MiB/s wr, 833 op/s 2024-02-10T05:42:15.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 41559 17.35 MB/sec warmup 69 sec latency 28.946 ms 2024-02-10T05:42:15.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:14 smithi192 ceph-mon[26904]: pgmap v273: 97 pgs: 97 active+clean; 250 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.8 MiB/s wr, 833 op/s 2024-02-10T05:42:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 42131 17.46 MB/sec warmup 70 sec latency 32.560 ms 2024-02-10T05:42:17.095 INFO:teuthology.orchestra.run.smithi028.stdout: 5 43033 17.64 MB/sec warmup 71 sec latency 19.326 ms 2024-02-10T05:42:17.111 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:16 smithi028 ceph-mon[24680]: pgmap v274: 97 pgs: 97 active+clean; 253 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.5 KiB/s rd, 8.2 MiB/s wr, 830 op/s 2024-02-10T05:42:17.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:16 smithi192 ceph-mon[26904]: pgmap v274: 97 pgs: 97 active+clean; 253 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.5 KiB/s rd, 8.2 MiB/s wr, 830 op/s 2024-02-10T05:42:18.096 INFO:teuthology.orchestra.run.smithi028.stdout: 5 43497 17.48 MB/sec warmup 72 sec latency 39.732 ms 2024-02-10T05:42:19.096 INFO:teuthology.orchestra.run.smithi028.stdout: 5 43852 17.30 MB/sec warmup 73 sec latency 46.531 ms 2024-02-10T05:42:19.266 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:19 smithi028 ceph-mon[24680]: pgmap v275: 97 pgs: 97 active+clean; 255 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.5 KiB/s rd, 8.7 MiB/s wr, 840 op/s 2024-02-10T05:42:19.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:19 smithi192 ceph-mon[26904]: pgmap v275: 97 pgs: 97 active+clean; 255 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.5 KiB/s rd, 8.7 MiB/s wr, 840 op/s 2024-02-10T05:42:20.096 INFO:teuthology.orchestra.run.smithi028.stdout: 5 44350 17.31 MB/sec warmup 74 sec latency 43.895 ms 2024-02-10T05:42:21.034 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:20 smithi028 ceph-mon[24680]: pgmap v276: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.6 MiB/s wr, 930 op/s 2024-02-10T05:42:21.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:20 smithi192 ceph-mon[26904]: pgmap v276: 97 pgs: 97 active+clean; 259 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.6 MiB/s wr, 930 op/s 2024-02-10T05:42:21.096 INFO:teuthology.orchestra.run.smithi028.stdout: 5 45084 17.22 MB/sec warmup 75 sec latency 32.609 ms 2024-02-10T05:42:22.096 INFO:teuthology.orchestra.run.smithi028.stdout: 5 45559 17.27 MB/sec warmup 76 sec latency 43.644 ms 2024-02-10T05:42:23.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:22 smithi028 ceph-mon[24680]: pgmap v277: 97 pgs: 97 active+clean; 260 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.2 MiB/s wr, 703 op/s 2024-02-10T05:42:23.101 INFO:teuthology.orchestra.run.smithi028.stdout: 5 46525 17.50 MB/sec warmup 77 sec latency 32.690 ms 2024-02-10T05:42:23.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:22 smithi192 ceph-mon[26904]: pgmap v277: 97 pgs: 97 active+clean; 260 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.2 MiB/s wr, 703 op/s 2024-02-10T05:42:24.101 INFO:teuthology.orchestra.run.smithi028.stdout: 5 47230 17.38 MB/sec warmup 78 sec latency 15.555 ms 2024-02-10T05:42:25.060 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:24 smithi028 ceph-mon[24680]: pgmap v278: 97 pgs: 97 active+clean; 268 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 916 op/s 2024-02-10T05:42:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: 5 48138 17.49 MB/sec warmup 79 sec latency 22.378 ms 2024-02-10T05:42:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:24 smithi192 ceph-mon[26904]: pgmap v278: 97 pgs: 97 active+clean; 268 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 916 op/s 2024-02-10T05:42:26.101 INFO:teuthology.orchestra.run.smithi028.stdout: 5 48781 17.42 MB/sec warmup 80 sec latency 26.511 ms 2024-02-10T05:42:27.101 INFO:teuthology.orchestra.run.smithi028.stdout: 5 49256 17.46 MB/sec warmup 81 sec latency 43.387 ms 2024-02-10T05:42:27.107 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:26 smithi028 ceph-mon[24680]: pgmap v279: 97 pgs: 97 active+clean; 275 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.3 MiB/s wr, 838 op/s 2024-02-10T05:42:27.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:26 smithi192 ceph-mon[26904]: pgmap v279: 97 pgs: 97 active+clean; 275 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.3 MiB/s wr, 838 op/s 2024-02-10T05:42:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: 5 49941 17.54 MB/sec warmup 82 sec latency 26.328 ms 2024-02-10T05:42:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: 5 50445 17.45 MB/sec warmup 83 sec latency 24.607 ms 2024-02-10T05:42:29.393 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:29 smithi028 ceph-mon[24680]: pgmap v280: 97 pgs: 97 active+clean; 278 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 809 op/s 2024-02-10T05:42:29.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:29 smithi192 ceph-mon[26904]: pgmap v280: 97 pgs: 97 active+clean; 278 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 809 op/s 2024-02-10T05:42:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: 5 51034 17.35 MB/sec warmup 84 sec latency 28.942 ms 2024-02-10T05:42:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:30 smithi192 ceph-mon[26904]: pgmap v281: 97 pgs: 97 active+clean; 286 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 964 op/s 2024-02-10T05:42:31.101 INFO:teuthology.orchestra.run.smithi028.stdout: 5 51790 17.38 MB/sec warmup 85 sec latency 21.741 ms 2024-02-10T05:42:31.168 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:30 smithi028 ceph-mon[24680]: pgmap v281: 97 pgs: 97 active+clean; 286 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 964 op/s 2024-02-10T05:42:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 52417 17.34 MB/sec warmup 86 sec latency 31.813 ms 2024-02-10T05:42:33.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 53088 17.43 MB/sec warmup 87 sec latency 30.005 ms 2024-02-10T05:42:33.187 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:32 smithi028 ceph-mon[24680]: pgmap v282: 97 pgs: 97 active+clean; 287 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 905 op/s 2024-02-10T05:42:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:32 smithi192 ceph-mon[26904]: pgmap v282: 97 pgs: 97 active+clean; 287 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 905 op/s 2024-02-10T05:42:34.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 53675 17.49 MB/sec warmup 88 sec latency 29.587 ms 2024-02-10T05:42:34.952 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:34 smithi028 ceph-mon[24680]: pgmap v283: 97 pgs: 97 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 947 op/s 2024-02-10T05:42:35.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 54228 17.37 MB/sec warmup 89 sec latency 29.164 ms 2024-02-10T05:42:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:34 smithi192 ceph-mon[26904]: pgmap v283: 97 pgs: 97 active+clean; 288 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 947 op/s 2024-02-10T05:42:36.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 54680 17.30 MB/sec warmup 90 sec latency 52.370 ms 2024-02-10T05:42:36.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:36 smithi028 ceph-mon[24680]: pgmap v284: 97 pgs: 97 active+clean; 292 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 795 op/s 2024-02-10T05:42:37.103 INFO:teuthology.orchestra.run.smithi028.stdout: 5 55414 17.30 MB/sec warmup 91 sec latency 22.583 ms 2024-02-10T05:42:37.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:36 smithi192 ceph-mon[26904]: pgmap v284: 97 pgs: 97 active+clean; 292 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 795 op/s 2024-02-10T05:42:38.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 56026 17.29 MB/sec warmup 92 sec latency 29.317 ms 2024-02-10T05:42:38.765 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:42:39.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 56591 17.33 MB/sec warmup 93 sec latency 33.045 ms 2024-02-10T05:42:39.254 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:39 smithi028 ceph-mon[24680]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:39.255 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:39 smithi028 ceph-mon[24680]: pgmap v285: 97 pgs: 97 active+clean; 297 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 748 op/s 2024-02-10T05:42:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:39 smithi192 ceph-mon[26904]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:39 smithi192 ceph-mon[26904]: pgmap v285: 97 pgs: 97 active+clean; 297 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 748 op/s 2024-02-10T05:42:39.653 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:42:39.653 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (6m) 95s ago 8m 24.2M - ba2b418f427c 110159ccf771 2024-02-10T05:42:39.653 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (7m) 95s ago 7m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:42:39.653 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (7m) 98s ago 7m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:42:39.653 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (6m) 95s ago 7m 51.7M - 8.3.5 dad864ee21e9 ea3f8ab1d935 2024-02-10T05:42:39.653 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (2m) 95s ago 2m 14.7M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:42:39.653 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (2m) 98s ago 2m 12.1M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:9283 running (9m) 95s ago 9m 460M - 17.2.0 e1d6a67b021e dc390c86451c 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283 running (7m) 98s ago 7m 414M - 17.2.0 e1d6a67b021e 5de705b3789d 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (9m) 95s ago 9m 43.5M 2048M 17.2.0 e1d6a67b021e a5294cba5afa 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (7m) 98s ago 7m 37.3M 2048M 17.2.0 e1d6a67b021e cf2f3de335d9 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (107s) 95s ago 106s 53.1M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (101s) 98s ago 100s 48.7M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (7m) 95s ago 7m 18.5M - 1dbe0e931976 07432e91f587 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (7m) 98s ago 7m 17.4M - 1dbe0e931976 54c7fecb7980 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (6m) 95s ago 6m 43.6M 3217M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (6m) 95s ago 6m 43.9M 3217M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (5m) 95s ago 5m 39.5M 3217M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (5m) 95s ago 5m 40.8M 3217M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:42:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (4m) 98s ago 4m 41.2M 3984M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:42:39.655 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (4m) 98s ago 4m 38.5M 3984M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:42:39.655 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (3m) 98s ago 3m 39.2M 3984M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:42:39.655 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (3m) 98s ago 3m 41.3M 3984M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:42:39.655 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (6m) 95s ago 6m 62.4M - 514e6a882f6e b9d9b936ea2c 2024-02-10T05:42:40.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 57152 17.38 MB/sec warmup 94 sec latency 25.447 ms 2024-02-10T05:42:40.206 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:42:40.206 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:42:40.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:42:40.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:42:40.206 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:42:40.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:42:40.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:42:40.206 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:42:40.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:42:40.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:42:40.206 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:42:40.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:42:40.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:42:40.207 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:42:40.207 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-02-10T05:42:40.207 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:42:40.207 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:42:40.258 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:40 smithi028 ceph-mon[24680]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:40.259 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:40 smithi028 ceph-mon[24680]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:40.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:40 smithi192 ceph-mon[26904]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:40.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:40 smithi192 ceph-mon[26904]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:40.606 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:42:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:42:40.607 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:42:40.607 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-02-10T05:42:40.607 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "0/23 daemons upgraded", 2024-02-10T05:42:40.607 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 image on host smithi192" 2024-02-10T05:42:40.607 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:42:41.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 57479 17.26 MB/sec warmup 95 sec latency 42.149 ms 2024-02-10T05:42:41.200 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:42:41.278 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:41 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/907669606' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:42:41.278 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:41 smithi028 ceph-mon[24680]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:41.278 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:41 smithi028 ceph-mon[24680]: pgmap v286: 97 pgs: 97 active+clean; 303 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.5 MiB/s wr, 879 op/s 2024-02-10T05:42:41.278 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:41 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/4062148787' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:42:41.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:41 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/907669606' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:42:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:41 smithi192 ceph-mon[26904]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:41 smithi192 ceph-mon[26904]: pgmap v286: 97 pgs: 97 active+clean; 303 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.5 MiB/s wr, 879 op/s 2024-02-10T05:42:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:41 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/4062148787' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:42:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 57869 17.11 MB/sec warmup 96 sec latency 48.415 ms 2024-02-10T05:42:43.086 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:42 smithi028 ceph-mon[24680]: pgmap v287: 97 pgs: 97 active+clean; 305 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 768 op/s 2024-02-10T05:42:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 58145 16.99 MB/sec warmup 97 sec latency 59.934 ms 2024-02-10T05:42:43.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:42 smithi192 ceph-mon[26904]: pgmap v287: 97 pgs: 97 active+clean; 305 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 768 op/s 2024-02-10T05:42:44.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 58718 17.02 MB/sec warmup 98 sec latency 44.844 ms 2024-02-10T05:42:45.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 59391 16.96 MB/sec warmup 99 sec latency 30.831 ms 2024-02-10T05:42:45.103 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:44 smithi028 ceph-mon[24680]: pgmap v288: 97 pgs: 97 active+clean; 309 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.4 MiB/s wr, 783 op/s 2024-02-10T05:42:45.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:44 smithi192 ceph-mon[26904]: pgmap v288: 97 pgs: 97 active+clean; 309 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.4 MiB/s wr, 783 op/s 2024-02-10T05:42:46.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 59819 16.98 MB/sec warmup 100 sec latency 45.038 ms 2024-02-10T05:42:47.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 60603 17.08 MB/sec warmup 101 sec latency 29.837 ms 2024-02-10T05:42:47.120 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:46 smithi028 ceph-mon[24680]: pgmap v289: 97 pgs: 97 active+clean; 311 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.3 MiB/s wr, 748 op/s 2024-02-10T05:42:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:46 smithi192 ceph-mon[26904]: pgmap v289: 97 pgs: 97 active+clean; 311 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.3 MiB/s wr, 748 op/s 2024-02-10T05:42:48.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 61115 17.01 MB/sec warmup 102 sec latency 30.506 ms 2024-02-10T05:42:49.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 61520 16.89 MB/sec warmup 103 sec latency 39.171 ms 2024-02-10T05:42:49.386 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:49 smithi028 ceph-mon[24680]: pgmap v290: 97 pgs: 97 active+clean; 311 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.2 MiB/s wr, 706 op/s 2024-02-10T05:42:49.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:49 smithi192 ceph-mon[26904]: pgmap v290: 97 pgs: 97 active+clean; 311 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.2 MiB/s wr, 706 op/s 2024-02-10T05:42:50.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 62020 16.89 MB/sec warmup 104 sec latency 27.024 ms 2024-02-10T05:42:51.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:50 smithi192 ceph-mon[26904]: pgmap v291: 97 pgs: 97 active+clean; 321 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.6 MiB/s wr, 873 op/s 2024-02-10T05:42:51.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 62762 16.88 MB/sec warmup 105 sec latency 29.455 ms 2024-02-10T05:42:51.150 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:50 smithi028 ceph-mon[24680]: pgmap v291: 97 pgs: 97 active+clean; 321 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.6 MiB/s wr, 873 op/s 2024-02-10T05:42:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 63243 16.86 MB/sec warmup 106 sec latency 25.071 ms 2024-02-10T05:42:53.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 63819 16.90 MB/sec warmup 107 sec latency 24.726 ms 2024-02-10T05:42:53.188 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:52 smithi028 ceph-mon[24680]: pgmap v292: 97 pgs: 97 active+clean; 321 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.3 MiB/s wr, 647 op/s 2024-02-10T05:42:53.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:52 smithi192 ceph-mon[26904]: pgmap v292: 97 pgs: 97 active+clean; 321 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.3 MiB/s wr, 647 op/s 2024-02-10T05:42:54.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 64270 16.90 MB/sec warmup 108 sec latency 99.150 ms 2024-02-10T05:42:54.958 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:54 smithi028 ceph-mon[24680]: pgmap v293: 97 pgs: 97 active+clean; 329 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.3 MiB/s wr, 876 op/s 2024-02-10T05:42:55.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 64752 16.81 MB/sec warmup 109 sec latency 19.464 ms 2024-02-10T05:42:55.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:54 smithi192 ceph-mon[26904]: pgmap v293: 97 pgs: 97 active+clean; 329 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.3 MiB/s wr, 876 op/s 2024-02-10T05:42:56.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 65200 16.72 MB/sec warmup 110 sec latency 38.529 ms 2024-02-10T05:42:56.729 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:56 smithi028 ceph-mon[24680]: Upgrade: Updating mgr.smithi192.hppzcm 2024-02-10T05:42:56.730 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:56 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:42:56.730 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:56 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.hppzcm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:42:56.730 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:56 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:42:56.730 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:56 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:56.730 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:56 smithi028 ceph-mon[24680]: Deploying daemon mgr.smithi192.hppzcm on smithi192 2024-02-10T05:42:57.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:56 smithi192 ceph-mon[26904]: Upgrade: Updating mgr.smithi192.hppzcm 2024-02-10T05:42:57.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:56 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:42:57.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:56 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.hppzcm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:42:57.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:56 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:42:57.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:56 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:57.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:56 smithi192 ceph-mon[26904]: Deploying daemon mgr.smithi192.hppzcm on smithi192 2024-02-10T05:42:57.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 65844 16.76 MB/sec warmup 111 sec latency 29.236 ms 2024-02-10T05:42:57.757 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:57 smithi028 ceph-mon[24680]: pgmap v294: 97 pgs: 97 active+clean; 333 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.9 MiB/s wr, 858 op/s 2024-02-10T05:42:57.954 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:57 smithi192 ceph-mon[26904]: pgmap v294: 97 pgs: 97 active+clean; 333 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.9 MiB/s wr, 858 op/s 2024-02-10T05:42:58.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 66455 16.70 MB/sec warmup 112 sec latency 58.725 ms 2024-02-10T05:42:59.103 INFO:teuthology.orchestra.run.smithi028.stdout: 5 66921 16.71 MB/sec warmup 113 sec latency 66.449 ms 2024-02-10T05:42:59.343 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:42:59 smithi028 ceph-mon[24680]: pgmap v295: 97 pgs: 97 active+clean; 336 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.8 MiB/s wr, 783 op/s 2024-02-10T05:42:59.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:42:59 smithi192 ceph-mon[26904]: pgmap v295: 97 pgs: 97 active+clean; 336 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.8 MiB/s wr, 783 op/s 2024-02-10T05:43:00.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 67468 16.72 MB/sec warmup 114 sec latency 78.817 ms 2024-02-10T05:43:01.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:00 smithi192 ceph-mon[26904]: pgmap v296: 97 pgs: 97 active+clean; 322 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.8 MiB/s wr, 929 op/s 2024-02-10T05:43:01.102 INFO:teuthology.orchestra.run.smithi028.stdout: 5 67820 16.70 MB/sec warmup 115 sec latency 130.825 ms 2024-02-10T05:43:01.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:00 smithi028 ceph-mon[24680]: pgmap v296: 97 pgs: 97 active+clean; 322 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 8.8 MiB/s wr, 929 op/s 2024-02-10T05:43:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: 5 68096 16.61 MB/sec warmup 116 sec latency 103.307 ms 2024-02-10T05:43:02.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:02 smithi028 ceph-mon[24680]: pgmap v297: 97 pgs: 97 active+clean; 316 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.2 MiB/s wr, 737 op/s 2024-02-10T05:43:03.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 68501 16.49 MB/sec warmup 117 sec latency 51.725 ms 2024-02-10T05:43:03.215 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:02 smithi192 ceph-mon[26904]: pgmap v297: 97 pgs: 97 active+clean; 316 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.2 MiB/s wr, 737 op/s 2024-02-10T05:43:04.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 69135 16.51 MB/sec warmup 118 sec latency 32.715 ms 2024-02-10T05:43:04.996 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:04 smithi192 ceph-mon[26904]: pgmap v298: 97 pgs: 97 active+clean; 330 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 930 op/s 2024-02-10T05:43:04.996 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:04 smithi028 ceph-mon[24680]: pgmap v298: 97 pgs: 97 active+clean; 330 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 930 op/s 2024-02-10T05:43:05.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 69959 16.52 MB/sec warmup 119 sec latency 40.176 ms 2024-02-10T05:43:06.264 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:43:06.264 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:06.264 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:06.264 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:06 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:06.537 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:43:06.537 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:06.537 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:06.537 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:06 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 71275 24.38 MB/sec execute 1 sec latency 54.031 ms 2024-02-10T05:43:07.187 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:07 smithi192 ceph-mon[26904]: pgmap v299: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.8 MiB/s wr, 688 op/s 2024-02-10T05:43:07.187 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:43:07.187 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:07 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:43:07.275 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:07 smithi028 ceph-mon[24680]: pgmap v299: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.8 MiB/s wr, 688 op/s 2024-02-10T05:43:07.276 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:43:07.276 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:07 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:43:08.110 INFO:teuthology.orchestra.run.smithi028.stdout: 5 71614 16.02 MB/sec execute 2 sec latency 117.810 ms 2024-02-10T05:43:09.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 72056 11.93 MB/sec execute 3 sec latency 39.385 ms 2024-02-10T05:43:09.294 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:09 smithi028 ceph-mon[24680]: pgmap v300: 97 pgs: 97 active+clean; 334 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.9 MiB/s wr, 625 op/s 2024-02-10T05:43:09.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:09 smithi192 ceph-mon[26904]: pgmap v300: 97 pgs: 97 active+clean; 334 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.9 MiB/s wr, 625 op/s 2024-02-10T05:43:10.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 72411 11.11 MB/sec execute 4 sec latency 26.104 ms 2024-02-10T05:43:10.855 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:10 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:43:10.856 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:10 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:43:10.982 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:10 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:43:10.983 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:10 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:43:11.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 72811 11.31 MB/sec execute 5 sec latency 46.867 ms 2024-02-10T05:43:11.653 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:43:11.866 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:11 smithi028 ceph-mon[24680]: pgmap v301: 97 pgs: 97 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.0 MiB/s wr, 752 op/s 2024-02-10T05:43:11.964 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:11 smithi192 ceph-mon[26904]: pgmap v301: 97 pgs: 97 active+clean; 331 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.0 MiB/s wr, 752 op/s 2024-02-10T05:43:12.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 73377 11.64 MB/sec execute 6 sec latency 100.230 ms 2024-02-10T05:43:12.705 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:43:12.705 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (7m) 2m ago 8m 24.2M - ba2b418f427c 110159ccf771 2024-02-10T05:43:12.705 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (8m) 2m ago 8m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:43:12.705 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (7m) 2s ago 7m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (6m) 2m ago 8m 51.7M - 8.3.5 dad864ee21e9 ea3f8ab1d935 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (2m) 2m ago 2m 14.7M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (2m) 2s ago 2m 12.6M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:9283 running (9m) 2m ago 9m 460M - 17.2.0 e1d6a67b021e dc390c86451c 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283 running (8s) 2s ago 7m 48.1M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddfed28689ae 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (9m) 2m ago 9m 43.5M 2048M 17.2.0 e1d6a67b021e a5294cba5afa 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (7m) 2s ago 7m 38.6M 2048M 17.2.0 e1d6a67b021e cf2f3de335d9 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (2m) 2m ago 2m 53.1M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (2m) 2s ago 2m 73.0M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (8m) 2m ago 8m 18.5M - 1dbe0e931976 07432e91f587 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (7m) 2s ago 7m 16.9M - 1dbe0e931976 54c7fecb7980 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (7m) 2m ago 7m 43.6M 3217M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (6m) 2m ago 6m 43.9M 3217M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (6m) 2m ago 6m 39.5M 3217M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:43:12.706 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (5m) 2m ago 5m 40.8M 3217M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:43:12.707 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (5m) 2s ago 5m 259M 3984M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:43:12.707 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (4m) 2s ago 4m 293M 3984M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:43:12.707 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (4m) 2s ago 4m 260M 3984M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:43:12.707 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (4m) 2s ago 4m 358M 3984M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:43:12.707 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (7m) 2m ago 7m 62.4M - 514e6a882f6e b9d9b936ea2c 2024-02-10T05:43:12.870 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:12 smithi028 ceph-mon[24680]: from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:13.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:12 smithi192 ceph-mon[26904]: from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:13.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 74031 12.27 MB/sec execute 7 sec latency 19.694 ms 2024-02-10T05:43:13.353 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:43:13.353 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:43:13.353 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:43:13.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:43:13.353 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:43:13.353 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:43:13.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:43:13.354 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:43:13.354 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:43:13.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:43:13.354 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:43:13.354 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:43:13.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:43:13.354 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:43:13.354 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-02-10T05:43:13.354 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:43:13.354 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:43:13.882 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:13 smithi028 ceph-mon[24680]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:13.882 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:13 smithi028 ceph-mon[24680]: from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:13.882 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:13 smithi028 ceph-mon[24680]: pgmap v302: 97 pgs: 97 active+clean; 320 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.1 MiB/s wr, 637 op/s 2024-02-10T05:43:13.882 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:13 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2904187714' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:43:13.882 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:13 smithi192 ceph-mon[26904]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:13.882 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:13 smithi192 ceph-mon[26904]: from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:13.882 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:13 smithi192 ceph-mon[26904]: pgmap v302: 97 pgs: 97 active+clean; 320 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 7.1 MiB/s wr, 637 op/s 2024-02-10T05:43:13.882 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:13 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2904187714' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:43:13.893 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:43:13.894 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:43:13.894 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:43:13.894 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-02-10T05:43:13.894 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "1/23 daemons upgraded", 2024-02-10T05:43:13.894 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading mgr daemons" 2024-02-10T05:43:13.894 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:43:14.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 74616 13.52 MB/sec execute 8 sec latency 17.957 ms 2024-02-10T05:43:14.503 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:43:14.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:14 smithi028 ceph-mon[24680]: from='client.24443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:14.895 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:14 smithi028 ceph-mon[24680]: Standby manager daemon smithi192.hppzcm restarted 2024-02-10T05:43:14.895 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:14 smithi028 ceph-mon[24680]: Standby manager daemon smithi192.hppzcm started 2024-02-10T05:43:14.895 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:14 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/crt"}]: dispatch 2024-02-10T05:43:14.895 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:14 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:43:14.895 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:14 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/key"}]: dispatch 2024-02-10T05:43:14.895 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:14 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:43:14.895 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:14 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2797320436' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:43:15.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:14 smithi192 ceph-mon[26904]: from='client.24443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:15.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:14 smithi192 ceph-mon[26904]: Standby manager daemon smithi192.hppzcm restarted 2024-02-10T05:43:15.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:14 smithi192 ceph-mon[26904]: Standby manager daemon smithi192.hppzcm started 2024-02-10T05:43:15.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:14 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/crt"}]: dispatch 2024-02-10T05:43:15.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:14 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:43:15.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:14 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/key"}]: dispatch 2024-02-10T05:43:15.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:14 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:43:15.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:14 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2797320436' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 75091 14.00 MB/sec execute 9 sec latency 41.515 ms 2024-02-10T05:43:15.902 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:15 smithi028 ceph-mon[24680]: mgrmap e22: smithi028.ucuqfx(active, since 8m), standbys: smithi192.hppzcm 2024-02-10T05:43:15.903 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:15 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:43:15.903 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:15 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:43:15.903 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:15 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:43:15.903 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:15 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:15.903 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:15 smithi028 ceph-mon[24680]: Upgrade: Need to upgrade myself (mgr.smithi028.ucuqfx) 2024-02-10T05:43:15.903 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:15 smithi028 ceph-mon[24680]: Failing over to other MGR 2024-02-10T05:43:15.903 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:15 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr fail", "who": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:43:15.903 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:15 smithi028 ceph-mon[24680]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T05:43:15.903 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:15 smithi028 ceph-mon[24680]: pgmap v304: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 747 op/s 2024-02-10T05:43:16.057 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:15 smithi192 ceph-mon[26904]: mgrmap e22: smithi028.ucuqfx(active, since 8m), standbys: smithi192.hppzcm 2024-02-10T05:43:16.057 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:15 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:43:16.057 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:15 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:43:16.057 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:15 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:43:16.057 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:15 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:16.057 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:15 smithi192 ceph-mon[26904]: Upgrade: Need to upgrade myself (mgr.smithi028.ucuqfx) 2024-02-10T05:43:16.057 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:15 smithi192 ceph-mon[26904]: Failing over to other MGR 2024-02-10T05:43:16.057 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:15 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr fail", "who": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:43:16.057 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:15 smithi192 ceph-mon[26904]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T05:43:16.057 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:15 smithi192 ceph-mon[26904]: pgmap v304: 97 pgs: 97 active+clean; 321 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 747 op/s 2024-02-10T05:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 75537 13.14 MB/sec execute 10 sec latency 54.953 ms 2024-02-10T05:43:16.908 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:43:16.908 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:43:16.908 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi028.xhtowe"}]: dispatch 2024-02-10T05:43:16.908 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi192.hdcbzw"}]: dispatch 2024-02-10T05:43:16.908 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mgr metadata", "who": "smithi192.hppzcm", "id": "smithi192.hppzcm"}]: dispatch 2024-02-10T05:43:16.908 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:43:16.908 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "mgr fail", "who": "smithi028.ucuqfx"}]': finished 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: mgrmap e23: smithi192.hppzcm(active, starting, since 0.996358s) 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: Manager daemon smithi192.hppzcm is now available 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:16.909 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:16.910 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:16.910 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:43:16.910 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.hppzcm/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:43:16.910 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.hppzcm/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:43:16.910 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.hppzcm/trash_purge_schedule"}]: dispatch 2024-02-10T05:43:16.910 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.hppzcm/trash_purge_schedule"}]: dispatch 2024-02-10T05:43:16.910 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:16.910 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:16 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:17.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 76117 13.18 MB/sec execute 11 sec latency 29.925 ms 2024-02-10T05:43:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:43:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:43:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi028.xhtowe"}]: dispatch 2024-02-10T05:43:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi192.hdcbzw"}]: dispatch 2024-02-10T05:43:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mgr metadata", "who": "smithi192.hppzcm", "id": "smithi192.hppzcm"}]: dispatch 2024-02-10T05:43:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:43:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:43:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:43:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:43:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:43:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:43:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:43:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.14202 172.21.15.28:0/3605289912' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "mgr fail", "who": "smithi028.ucuqfx"}]': finished 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: mgrmap e23: smithi192.hppzcm(active, starting, since 0.996358s) 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: Manager daemon smithi192.hppzcm is now available 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.hppzcm/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.hppzcm/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.hppzcm/trash_purge_schedule"}]: dispatch 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi192.hppzcm/trash_purge_schedule"}]: dispatch 2024-02-10T05:43:17.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:17.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:16 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:17.911 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:17 smithi028 ceph-mon[24680]: Deploying cephadm binary to smithi192 2024-02-10T05:43:17.911 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:17 smithi028 ceph-mon[24680]: Deploying cephadm binary to smithi028 2024-02-10T05:43:17.912 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:17 smithi028 ceph-mon[24680]: [10/Feb/2024:05:43:16] ENGINE Bus STARTING 2024-02-10T05:43:17.912 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:17 smithi028 ceph-mon[24680]: mgrmap e24: smithi192.hppzcm(active, since 2s) 2024-02-10T05:43:17.912 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:17 smithi028 ceph-mon[24680]: pgmap v3: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-02-10T05:43:17.912 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:17 smithi028 ceph-mon[24680]: [10/Feb/2024:05:43:17] ENGINE Serving on https://172.21.15.192:7150 2024-02-10T05:43:18.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 76858 13.77 MB/sec execute 12 sec latency 32.366 ms 2024-02-10T05:43:18.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:17 smithi192 ceph-mon[26904]: Deploying cephadm binary to smithi192 2024-02-10T05:43:18.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:17 smithi192 ceph-mon[26904]: Deploying cephadm binary to smithi028 2024-02-10T05:43:18.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:17 smithi192 ceph-mon[26904]: [10/Feb/2024:05:43:16] ENGINE Bus STARTING 2024-02-10T05:43:18.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:17 smithi192 ceph-mon[26904]: mgrmap e24: smithi192.hppzcm(active, since 2s) 2024-02-10T05:43:18.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:17 smithi192 ceph-mon[26904]: pgmap v3: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-02-10T05:43:18.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:17 smithi192 ceph-mon[26904]: [10/Feb/2024:05:43:17] ENGINE Serving on https://172.21.15.192:7150 2024-02-10T05:43:18.917 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:18 smithi028 ceph-mon[24680]: [10/Feb/2024:05:43:17] ENGINE Serving on http://172.21.15.192:8765 2024-02-10T05:43:18.917 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:18 smithi028 ceph-mon[24680]: [10/Feb/2024:05:43:17] ENGINE Bus STARTED 2024-02-10T05:43:18.917 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:18 smithi028 ceph-mon[24680]: pgmap v4: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-02-10T05:43:19.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 77621 14.10 MB/sec execute 13 sec latency 25.121 ms 2024-02-10T05:43:19.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:18 smithi192 ceph-mon[26904]: [10/Feb/2024:05:43:17] ENGINE Serving on http://172.21.15.192:8765 2024-02-10T05:43:19.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:18 smithi192 ceph-mon[26904]: [10/Feb/2024:05:43:17] ENGINE Bus STARTED 2024-02-10T05:43:19.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:18 smithi192 ceph-mon[26904]: pgmap v4: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-02-10T05:43:19.944 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:19 smithi028 ceph-mon[24680]: mgrmap e25: smithi192.hppzcm(active, since 4s) 2024-02-10T05:43:20.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 78314 15.04 MB/sec execute 14 sec latency 28.037 ms 2024-02-10T05:43:20.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:19 smithi192 ceph-mon[26904]: mgrmap e25: smithi192.hppzcm(active, since 4s) 2024-02-10T05:43:20.948 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:20 smithi028 ceph-mon[24680]: pgmap v5: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-02-10T05:43:21.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:20 smithi192 ceph-mon[26904]: pgmap v5: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-02-10T05:43:21.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 78806 14.92 MB/sec execute 15 sec latency 23.999 ms 2024-02-10T05:43:21.958 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:21 smithi028 ceph-mon[24680]: mgrmap e26: smithi192.hppzcm(active, since 6s) 2024-02-10T05:43:22.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 79143 14.32 MB/sec execute 16 sec latency 43.511 ms 2024-02-10T05:43:22.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:21 smithi192 ceph-mon[26904]: mgrmap e26: smithi192.hppzcm(active, since 6s) 2024-02-10T05:43:22.965 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:22 smithi028 ceph-mon[24680]: pgmap v6: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-02-10T05:43:23.108 INFO:teuthology.orchestra.run.smithi028.stdout: 5 79565 14.10 MB/sec execute 17 sec latency 47.132 ms 2024-02-10T05:43:23.222 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:22 smithi192 ceph-mon[26904]: pgmap v6: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-02-10T05:43:24.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 80037 13.97 MB/sec execute 18 sec latency 50.390 ms 2024-02-10T05:43:24.234 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:24 smithi028 ceph-mon[24680]: pgmap v7: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 6.4 MiB/s wr, 619 op/s 2024-02-10T05:43:24.536 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:24 smithi192 ceph-mon[26904]: pgmap v7: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 6.4 MiB/s wr, 619 op/s 2024-02-10T05:43:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 80485 13.79 MB/sec execute 19 sec latency 43.038 ms 2024-02-10T05:43:25.247 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:25 smithi028 ceph-mon[24680]: Standby manager daemon smithi028.ucuqfx started 2024-02-10T05:43:25.248 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:25 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.28:0/914360067' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi028.ucuqfx/crt"}]: dispatch 2024-02-10T05:43:25.248 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:25 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.28:0/914360067' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:43:25.248 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:25 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.28:0/914360067' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi028.ucuqfx/key"}]: dispatch 2024-02-10T05:43:25.248 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:25 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.28:0/914360067' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:43:25.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:25 smithi192 ceph-mon[26904]: Standby manager daemon smithi028.ucuqfx started 2024-02-10T05:43:25.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:25 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.28:0/914360067' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi028.ucuqfx/crt"}]: dispatch 2024-02-10T05:43:25.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:25 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.28:0/914360067' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:43:25.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:25 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.28:0/914360067' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi028.ucuqfx/key"}]: dispatch 2024-02-10T05:43:25.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:25 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.28:0/914360067' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:43:26.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 81026 13.77 MB/sec execute 20 sec latency 59.825 ms 2024-02-10T05:43:26.258 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:26 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mgr metadata", "who": "smithi028.ucuqfx", "id": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:43:26.259 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:26 smithi028 ceph-mon[24680]: mgrmap e27: smithi192.hppzcm(active, since 10s), standbys: smithi028.ucuqfx 2024-02-10T05:43:26.259 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:26 smithi028 ceph-mon[24680]: pgmap v8: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 5.0 MiB/s wr, 481 op/s 2024-02-10T05:43:26.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:26 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mgr metadata", "who": "smithi028.ucuqfx", "id": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:43:26.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:26 smithi192 ceph-mon[26904]: mgrmap e27: smithi192.hppzcm(active, since 10s), standbys: smithi028.ucuqfx 2024-02-10T05:43:26.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:26 smithi192 ceph-mon[26904]: pgmap v8: 97 pgs: 97 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 5.0 MiB/s wr, 481 op/s 2024-02-10T05:43:27.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 81479 13.85 MB/sec execute 21 sec latency 102.004 ms 2024-02-10T05:43:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 81950 14.05 MB/sec execute 22 sec latency 32.940 ms 2024-02-10T05:43:29.041 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:28 smithi028 ceph-mon[24680]: pgmap v9: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 7.0 MiB/s wr, 688 op/s 2024-02-10T05:43:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 82372 13.86 MB/sec execute 23 sec latency 37.806 ms 2024-02-10T05:43:29.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:28 smithi192 ceph-mon[26904]: pgmap v9: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 7.0 MiB/s wr, 688 op/s 2024-02-10T05:43:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 82752 13.55 MB/sec execute 24 sec latency 78.391 ms 2024-02-10T05:43:31.061 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:30 smithi028 ceph-mon[24680]: pgmap v10: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 629 op/s 2024-02-10T05:43:31.062 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:30 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:31.062 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:30 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:31.062 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:30 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:31.062 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:30 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:31.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:30 smithi192 ceph-mon[26904]: pgmap v10: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 629 op/s 2024-02-10T05:43:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:30 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:30 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:30 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:30 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:31.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 83310 13.63 MB/sec execute 25 sec latency 26.642 ms 2024-02-10T05:43:32.066 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:43:32.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:32.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:32.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:32.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:32.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.067 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.068 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.068 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:31 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:32.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 83903 13.69 MB/sec execute 26 sec latency 30.971 ms 2024-02-10T05:43:32.251 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:43:32.251 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:32.251 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:32.251 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:32.251 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:32.251 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.251 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.251 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.251 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.252 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.252 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.252 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.252 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:32.252 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:31 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:33.084 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:32 smithi028 ceph-mon[24680]: Adjusting osd_memory_target on smithi028 to 1937M 2024-02-10T05:43:33.084 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:32 smithi028 ceph-mon[24680]: pgmap v11: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 629 op/s 2024-02-10T05:43:33.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 84523 13.69 MB/sec execute 27 sec latency 43.864 ms 2024-02-10T05:43:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:32 smithi192 ceph-mon[26904]: Adjusting osd_memory_target on smithi028 to 1937M 2024-02-10T05:43:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:32 smithi192 ceph-mon[26904]: pgmap v11: 97 pgs: 97 active+clean; 329 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 629 op/s 2024-02-10T05:43:34.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 85124 13.95 MB/sec execute 28 sec latency 23.043 ms 2024-02-10T05:43:34.363 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:34 smithi028 ceph-mon[24680]: pgmap v12: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 9.4 MiB/s wr, 916 op/s 2024-02-10T05:43:34.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:34 smithi192 ceph-mon[26904]: pgmap v12: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 9.4 MiB/s wr, 916 op/s 2024-02-10T05:43:35.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 85678 14.10 MB/sec execute 29 sec latency 41.927 ms 2024-02-10T05:43:36.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 86132 13.94 MB/sec execute 30 sec latency 47.944 ms 2024-02-10T05:43:36.887 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:36.887 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:36.887 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:36.887 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:36.887 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:36.887 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:36.887 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:36.887 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:36.887 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:36.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:36.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: Adjusting osd_memory_target on smithi192 to 2704M 2024-02-10T05:43:36.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:36.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:36.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:36.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-10T05:43:36.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: Updating smithi192:/etc/ceph/ceph.conf 2024-02-10T05:43:36.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: pgmap v13: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.6 MiB/s wr, 556 op/s 2024-02-10T05:43:36.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:36 smithi028 ceph-mon[24680]: Updating smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:37.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:37.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:37.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:37.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: Adjusting osd_memory_target on smithi192 to 2704M 2024-02-10T05:43:37.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:37.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:37.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:37.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-10T05:43:37.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: Updating smithi192:/etc/ceph/ceph.conf 2024-02-10T05:43:37.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: pgmap v13: 97 pgs: 97 active+clean; 330 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.6 MiB/s wr, 556 op/s 2024-02-10T05:43:37.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:36 smithi192 ceph-mon[26904]: Updating smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 86586 13.88 MB/sec execute 31 sec latency 30.966 ms 2024-02-10T05:43:37.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: Updating smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:37.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:37.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:37.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: Updating smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:37.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: Updating smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:37.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:37.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:37.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:37.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:37.894 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:37.895 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:37.895 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:37.895 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:37.895 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:37.895 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:37 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:38.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 87105 13.82 MB/sec execute 32 sec latency 33.935 ms 2024-02-10T05:43:38.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: Updating smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:38.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:38.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:38.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: Updating smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:38.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: Updating smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:38.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:38.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:38.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:38.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:38.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:38.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:38.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:38.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:38.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:38.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:37 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:38.897 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:38 smithi028 ceph-mon[24680]: Reconfiguring alertmanager.smithi028 (dependencies changed)... 2024-02-10T05:43:38.898 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:38 smithi028 ceph-mon[24680]: Reconfiguring daemon alertmanager.smithi028 on smithi028 2024-02-10T05:43:38.898 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:38 smithi028 ceph-mon[24680]: pgmap v14: 97 pgs: 97 active+clean; 336 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 881 op/s 2024-02-10T05:43:39.114 INFO:teuthology.orchestra.run.smithi028.stdout: 5 87593 13.75 MB/sec execute 33 sec latency 35.524 ms 2024-02-10T05:43:39.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:38 smithi192 ceph-mon[26904]: Reconfiguring alertmanager.smithi028 (dependencies changed)... 2024-02-10T05:43:39.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:38 smithi192 ceph-mon[26904]: Reconfiguring daemon alertmanager.smithi028 on smithi028 2024-02-10T05:43:39.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:38 smithi192 ceph-mon[26904]: pgmap v14: 97 pgs: 97 active+clean; 336 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 881 op/s 2024-02-10T05:43:40.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 88144 13.73 MB/sec execute 34 sec latency 43.150 ms 2024-02-10T05:43:41.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:40 smithi192 ceph-mon[26904]: pgmap v15: 97 pgs: 97 active+clean; 336 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.5 MiB/s wr, 611 op/s 2024-02-10T05:43:41.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 88583 13.78 MB/sec execute 35 sec latency 28.361 ms 2024-02-10T05:43:41.175 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:40 smithi028 ceph-mon[24680]: pgmap v15: 97 pgs: 97 active+clean; 336 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.5 MiB/s wr, 611 op/s 2024-02-10T05:43:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 88998 13.85 MB/sec execute 36 sec latency 36.477 ms 2024-02-10T05:43:43.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:42 smithi028 ceph-mon[24680]: pgmap v16: 97 pgs: 97 active+clean; 336 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.5 MiB/s wr, 611 op/s 2024-02-10T05:43:43.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 89485 13.80 MB/sec execute 37 sec latency 41.001 ms 2024-02-10T05:43:43.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:42 smithi192 ceph-mon[26904]: pgmap v16: 97 pgs: 97 active+clean; 336 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.5 MiB/s wr, 611 op/s 2024-02-10T05:43:44.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 89886 13.69 MB/sec execute 38 sec latency 73.859 ms 2024-02-10T05:43:44.317 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:44 smithi028 ceph-mon[24680]: pgmap v17: 97 pgs: 97 active+clean; 334 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 898 op/s 2024-02-10T05:43:44.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:44 smithi192 ceph-mon[26904]: pgmap v17: 97 pgs: 97 active+clean; 334 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 898 op/s 2024-02-10T05:43:45.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 90399 13.64 MB/sec execute 39 sec latency 22.048 ms 2024-02-10T05:43:45.139 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:43:46.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 90911 13.60 MB/sec execute 40 sec latency 35.321 ms 2024-02-10T05:43:46.297 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:43:46.297 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (7m) 16s ago 9m 27.6M - 0.23.0 ba2b418f427c 110159ccf771 2024-02-10T05:43:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (9m) 16s ago 9m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:43:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (8m) 15s ago 8m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:43:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (7m) 16s ago 8m 52.3M - 8.3.5 dad864ee21e9 ea3f8ab1d935 2024-02-10T05:43:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (3m) 16s ago 3m 36.5M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:43:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (3m) 15s ago 3m 13.0M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:43:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:9283 running (10m) 16s ago 10m 333M - 17.2.0 e1d6a67b021e dc390c86451c 2024-02-10T05:43:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283 running (41s) 15s ago 8m 488M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddfed28689ae 2024-02-10T05:43:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (10m) 16s ago 10m 49.3M 2048M 17.2.0 e1d6a67b021e a5294cba5afa 2024-02-10T05:43:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (8m) 15s ago 8m 40.4M 2048M 17.2.0 e1d6a67b021e cf2f3de335d9 2024-02-10T05:43:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (2m) 16s ago 2m 262M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:43:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (2m) 15s ago 2m 73.3M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:43:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (8m) 16s ago 8m 21.0M - 1.3.1 1dbe0e931976 07432e91f587 2024-02-10T05:43:46.299 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (8m) 15s ago 8m 19.3M - 1.3.1 1dbe0e931976 54c7fecb7980 2024-02-10T05:43:46.299 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (7m) 16s ago 7m 326M 1937M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:43:46.299 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (7m) 16s ago 7m 255M 1937M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:43:46.299 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (6m) 16s ago 6m 207M 1937M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:43:46.299 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (6m) 16s ago 6m 267M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:43:46.299 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (5m) 15s ago 5m 308M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:43:46.299 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (5m) 15s ago 5m 321M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:43:46.299 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (5m) 15s ago 5m 264M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:43:46.299 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (4m) 15s ago 4m 396M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:43:46.299 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (7m) 16s ago 7m 64.6M - 2.33.4 514e6a882f6e b9d9b936ea2c 2024-02-10T05:43:46.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:46 smithi028 ceph-mon[24680]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:46.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:46 smithi028 ceph-mon[24680]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:46.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:46 smithi028 ceph-mon[24680]: pgmap v18: 97 pgs: 97 active+clean; 334 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.5 MiB/s wr, 611 op/s 2024-02-10T05:43:46.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:46 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:43:47.025 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:43:47.025 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:43:47.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:43:47.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:43:47.025 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:43:47.025 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:43:47.026 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:43:47.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 91404 13.53 MB/sec execute 41 sec latency 45.687 ms 2024-02-10T05:43:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:46 smithi192 ceph-mon[26904]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:46 smithi192 ceph-mon[26904]: from='client.14738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:46 smithi192 ceph-mon[26904]: pgmap v18: 97 pgs: 97 active+clean; 334 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.5 MiB/s wr, 611 op/s 2024-02-10T05:43:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:46 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:43:47.606 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:43:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:43:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:43:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:43:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-02-10T05:43:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "1/23 daemons upgraded", 2024-02-10T05:43:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-02-10T05:43:47.606 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:43:47.606 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:43:48.093 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:47 smithi028 ceph-mon[24680]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:48.093 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:47 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2531434535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:43:48.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 91969 13.58 MB/sec execute 42 sec latency 32.340 ms 2024-02-10T05:43:48.195 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:43:48.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:47 smithi192 ceph-mon[26904]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:48.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:47 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2531434535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:43:49.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 92506 13.80 MB/sec execute 43 sec latency 31.605 ms 2024-02-10T05:43:49.122 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:48 smithi028 ceph-mon[24680]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:49.122 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:48 smithi028 ceph-mon[24680]: pgmap v19: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 898 op/s 2024-02-10T05:43:49.122 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:48 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1293464407' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:43:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:48 smithi192 ceph-mon[26904]: from='client.14750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:48 smithi192 ceph-mon[26904]: pgmap v19: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 898 op/s 2024-02-10T05:43:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:48 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/1293464407' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:43:50.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 92991 13.75 MB/sec execute 44 sec latency 32.710 ms 2024-02-10T05:43:51.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:50 smithi192 ceph-mon[26904]: pgmap v20: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.6 MiB/s wr, 573 op/s 2024-02-10T05:43:51.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 93349 13.64 MB/sec execute 45 sec latency 46.115 ms 2024-02-10T05:43:51.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:50 smithi028 ceph-mon[24680]: pgmap v20: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.6 MiB/s wr, 573 op/s 2024-02-10T05:43:52.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 93656 13.46 MB/sec execute 46 sec latency 48.960 ms 2024-02-10T05:43:53.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 94086 13.43 MB/sec execute 47 sec latency 41.943 ms 2024-02-10T05:43:53.192 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:52 smithi028 ceph-mon[24680]: pgmap v21: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.6 MiB/s wr, 573 op/s 2024-02-10T05:43:53.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:52 smithi192 ceph-mon[26904]: pgmap v21: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.6 MiB/s wr, 573 op/s 2024-02-10T05:43:54.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 94505 13.35 MB/sec execute 48 sec latency 66.718 ms 2024-02-10T05:43:54.460 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:54 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:54.460 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:54 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:54.461 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:54 smithi028 ceph-mon[24680]: Reconfiguring grafana.smithi028 (dependencies changed)... 2024-02-10T05:43:54.461 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:54 smithi028 ceph-mon[24680]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T05:43:54.461 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:54 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:54.461 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:54 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:54.461 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:54 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:43:54.461 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:54 smithi028 ceph-mon[24680]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:43:54.461 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:54 smithi028 ceph-mon[24680]: Reconfiguring daemon grafana.smithi028 on smithi028 2024-02-10T05:43:54.461 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:54 smithi028 ceph-mon[24680]: pgmap v22: 97 pgs: 97 active+clean; 337 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 849 op/s 2024-02-10T05:43:54.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:54 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:54.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:54 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:54.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:54 smithi192 ceph-mon[26904]: Reconfiguring grafana.smithi028 (dependencies changed)... 2024-02-10T05:43:54.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:54 smithi192 ceph-mon[26904]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T05:43:54.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:54 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:54.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:54 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:43:54.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:54 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:43:54.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:54 smithi192 ceph-mon[26904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:43:54.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:54 smithi192 ceph-mon[26904]: Reconfiguring daemon grafana.smithi028 on smithi028 2024-02-10T05:43:54.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:54 smithi192 ceph-mon[26904]: pgmap v22: 97 pgs: 97 active+clean; 337 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 849 op/s 2024-02-10T05:43:55.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 94935 13.27 MB/sec execute 49 sec latency 37.866 ms 2024-02-10T05:43:56.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 95535 13.32 MB/sec execute 50 sec latency 24.838 ms 2024-02-10T05:43:56.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:56 smithi028 ceph-mon[24680]: pgmap v23: 97 pgs: 97 active+clean; 337 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 562 op/s 2024-02-10T05:43:57.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 96081 13.52 MB/sec execute 51 sec latency 35.240 ms 2024-02-10T05:43:57.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:56 smithi192 ceph-mon[26904]: pgmap v23: 97 pgs: 97 active+clean; 337 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 562 op/s 2024-02-10T05:43:58.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 96569 13.49 MB/sec execute 52 sec latency 37.081 ms 2024-02-10T05:43:58.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:43:58 smithi028 ceph-mon[24680]: pgmap v24: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 849 op/s 2024-02-10T05:43:59.116 INFO:teuthology.orchestra.run.smithi028.stdout: 5 96917 13.38 MB/sec execute 53 sec latency 36.004 ms 2024-02-10T05:43:59.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:43:58 smithi192 ceph-mon[26904]: pgmap v24: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 849 op/s 2024-02-10T05:44:00.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 97304 13.36 MB/sec execute 54 sec latency 46.500 ms 2024-02-10T05:44:01.004 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:00 smithi028 ceph-mon[24680]: pgmap v25: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.0 MiB/s wr, 562 op/s 2024-02-10T05:44:01.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:00 smithi192 ceph-mon[26904]: pgmap v25: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.0 MiB/s wr, 562 op/s 2024-02-10T05:44:01.115 INFO:teuthology.orchestra.run.smithi028.stdout: 5 97647 13.23 MB/sec execute 55 sec latency 57.796 ms 2024-02-10T05:44:02.011 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:01 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:02.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 98008 13.13 MB/sec execute 56 sec latency 55.165 ms 2024-02-10T05:44:02.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:01 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:03.021 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:02 smithi028 ceph-mon[24680]: pgmap v26: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.0 MiB/s wr, 562 op/s 2024-02-10T05:44:03.119 INFO:teuthology.orchestra.run.smithi028.stdout: 5 98341 13.02 MB/sec execute 57 sec latency 35.062 ms 2024-02-10T05:44:03.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:02 smithi192 ceph-mon[26904]: pgmap v26: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.0 MiB/s wr, 562 op/s 2024-02-10T05:44:04.119 INFO:teuthology.orchestra.run.smithi028.stdout: 5 98846 13.01 MB/sec execute 58 sec latency 31.008 ms 2024-02-10T05:44:04.284 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:04 smithi028 ceph-mon[24680]: pgmap v27: 97 pgs: 97 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 802 op/s 2024-02-10T05:44:04.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:04 smithi192 ceph-mon[26904]: pgmap v27: 97 pgs: 97 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 802 op/s 2024-02-10T05:44:05.119 INFO:teuthology.orchestra.run.smithi028.stdout: 5 99367 13.16 MB/sec execute 59 sec latency 33.034 ms 2024-02-10T05:44:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 99679 13.10 MB/sec execute 60 sec latency 49.514 ms 2024-02-10T05:44:07.094 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:06 smithi028 ceph-mon[24680]: pgmap v28: 97 pgs: 97 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.1 MiB/s wr, 526 op/s 2024-02-10T05:44:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: 5 100007 13.01 MB/sec execute 61 sec latency 45.280 ms 2024-02-10T05:44:07.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:06 smithi192 ceph-mon[26904]: pgmap v28: 97 pgs: 97 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.1 MiB/s wr, 526 op/s 2024-02-10T05:44:08.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 100334 12.92 MB/sec execute 62 sec latency 45.350 ms 2024-02-10T05:44:09.119 INFO:teuthology.orchestra.run.smithi028.stdout: 5 100718 12.87 MB/sec execute 63 sec latency 37.712 ms 2024-02-10T05:44:09.166 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:08 smithi028 ceph-mon[24680]: pgmap v29: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 765 op/s 2024-02-10T05:44:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:08 smithi192 ceph-mon[26904]: pgmap v29: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 765 op/s 2024-02-10T05:44:10.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 101151 12.85 MB/sec execute 64 sec latency 82.828 ms 2024-02-10T05:44:11.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:10 smithi192 ceph-mon[26904]: pgmap v30: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 478 op/s 2024-02-10T05:44:11.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 101619 12.81 MB/sec execute 65 sec latency 38.462 ms 2024-02-10T05:44:11.196 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:10 smithi028 ceph-mon[24680]: pgmap v30: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 478 op/s 2024-02-10T05:44:12.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 101984 12.71 MB/sec execute 66 sec latency 50.654 ms 2024-02-10T05:44:12.964 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:12 smithi028 ceph-mon[24680]: pgmap v31: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 478 op/s 2024-02-10T05:44:13.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 102422 12.70 MB/sec execute 67 sec latency 44.074 ms 2024-02-10T05:44:13.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:12 smithi192 ceph-mon[26904]: pgmap v31: 97 pgs: 97 active+clean; 331 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 478 op/s 2024-02-10T05:44:14.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 102797 12.75 MB/sec execute 68 sec latency 38.759 ms 2024-02-10T05:44:14.476 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:14 smithi028 ceph-mon[24680]: pgmap v32: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 722 op/s 2024-02-10T05:44:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:14 smithi192 ceph-mon[26904]: pgmap v32: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 722 op/s 2024-02-10T05:44:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 103294 12.78 MB/sec execute 69 sec latency 31.934 ms 2024-02-10T05:44:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 103730 12.77 MB/sec execute 70 sec latency 54.238 ms 2024-02-10T05:44:17.007 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:16 smithi028 ceph-mon[24680]: pgmap v33: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.9 MiB/s wr, 482 op/s 2024-02-10T05:44:17.008 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:17.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 104053 12.72 MB/sec execute 71 sec latency 59.086 ms 2024-02-10T05:44:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:16 smithi192 ceph-mon[26904]: pgmap v33: 97 pgs: 97 active+clean; 328 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.9 MiB/s wr, 482 op/s 2024-02-10T05:44:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:18.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 104461 12.68 MB/sec execute 72 sec latency 59.325 ms 2024-02-10T05:44:18.755 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:44:19.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:18 smithi028 ceph-mon[24680]: pgmap v34: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.7 MiB/s wr, 734 op/s 2024-02-10T05:44:19.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 104800 12.59 MB/sec execute 73 sec latency 56.869 ms 2024-02-10T05:44:19.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:18 smithi192 ceph-mon[26904]: pgmap v34: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.7 MiB/s wr, 734 op/s 2024-02-10T05:44:19.790 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:44:19.790 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 starting - - - - 2024-02-10T05:44:19.790 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (9m) 49s ago 9m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:44:19.790 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (9m) 49s ago 9m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:44:19.790 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (8m) 49s ago 9m 52.3M - 8.3.5 dad864ee21e9 ea3f8ab1d935 2024-02-10T05:44:19.790 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (3m) 49s ago 3m 36.5M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (3m) 49s ago 3m 13.0M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:9283 running (10m) 49s ago 10m 333M - 17.2.0 e1d6a67b021e dc390c86451c 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283 running (75s) 49s ago 8m 488M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddfed28689ae 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (11m) 49s ago 11m 49.3M 2048M 17.2.0 e1d6a67b021e a5294cba5afa 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (8m) 49s ago 8m 40.4M 2048M 17.2.0 e1d6a67b021e cf2f3de335d9 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (3m) 49s ago 3m 262M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (3m) 49s ago 3m 73.3M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (9m) 49s ago 9m 21.0M - 1.3.1 1dbe0e931976 07432e91f587 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (8m) 49s ago 8m 19.3M - 1.3.1 1dbe0e931976 54c7fecb7980 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (8m) 49s ago 8m 326M 1937M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (7m) 49s ago 7m 255M 1937M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (7m) 49s ago 7m 207M 1937M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:44:19.791 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (6m) 49s ago 6m 267M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:44:19.792 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (6m) 49s ago 6m 308M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:44:19.792 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (6m) 49s ago 6m 321M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:44:19.792 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (5m) 49s ago 5m 264M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:44:19.792 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (5m) 49s ago 5m 396M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:44:19.792 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (8m) 49s ago 8m 64.6M - 2.33.4 514e6a882f6e b9d9b936ea2c 2024-02-10T05:44:20.040 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:19 smithi028 ceph-mon[24680]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:20.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 105110 12.50 MB/sec execute 74 sec latency 77.902 ms 2024-02-10T05:44:20.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:19 smithi192 ceph-mon[26904]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:20.444 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:44:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:44:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:44:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:44:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:44:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T05:44:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T05:44:20.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:44:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:44:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:44:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:44:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:44:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:44:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:44:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:44:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T05:44:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T05:44:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:44:20.445 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:44:20.973 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:44:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:44:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:44:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:44:20.973 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-02-10T05:44:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "1/23 daemons upgraded", 2024-02-10T05:44:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-02-10T05:44:20.974 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:44:20.974 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:44:21.048 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:20 smithi028 ceph-mon[24680]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:21.048 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:20 smithi028 ceph-mon[24680]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:21.049 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:20 smithi028 ceph-mon[24680]: pgmap v35: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.1 MiB/s wr, 495 op/s 2024-02-10T05:44:21.049 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:20 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/132932438' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:44:21.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:20 smithi192 ceph-mon[26904]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:20 smithi192 ceph-mon[26904]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:20 smithi192 ceph-mon[26904]: pgmap v35: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.1 MiB/s wr, 495 op/s 2024-02-10T05:44:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:20 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/132932438' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:44:21.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 105420 12.44 MB/sec execute 75 sec latency 99.739 ms 2024-02-10T05:44:21.612 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:44:22.056 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:21 smithi028 ceph-mon[24680]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:22.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:21 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1758050488' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:44:22.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:21 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:22.057 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:21 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:22.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 105830 12.42 MB/sec execute 76 sec latency 43.943 ms 2024-02-10T05:44:22.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:21 smithi192 ceph-mon[26904]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:22.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:21 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/1758050488' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:44:22.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:21 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:22.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:21 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:23.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:22 smithi028 ceph-mon[24680]: Reconfiguring node-exporter.smithi028 (dependencies changed)... 2024-02-10T05:44:23.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:22 smithi028 ceph-mon[24680]: Reconfiguring daemon node-exporter.smithi028 on smithi028 2024-02-10T05:44:23.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:22 smithi028 ceph-mon[24680]: pgmap v36: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.1 MiB/s wr, 495 op/s 2024-02-10T05:44:23.120 INFO:teuthology.orchestra.run.smithi028.stdout: 5 106357 12.51 MB/sec execute 77 sec latency 54.088 ms 2024-02-10T05:44:23.222 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:22 smithi192 ceph-mon[26904]: Reconfiguring node-exporter.smithi028 (dependencies changed)... 2024-02-10T05:44:23.222 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:22 smithi192 ceph-mon[26904]: Reconfiguring daemon node-exporter.smithi028 on smithi028 2024-02-10T05:44:23.222 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:22 smithi192 ceph-mon[26904]: pgmap v36: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.1 MiB/s wr, 495 op/s 2024-02-10T05:44:24.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 107041 12.61 MB/sec execute 78 sec latency 26.583 ms 2024-02-10T05:44:24.334 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:24 smithi028 ceph-mon[24680]: pgmap v37: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 707 op/s 2024-02-10T05:44:24.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:24 smithi192 ceph-mon[26904]: pgmap v37: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 707 op/s 2024-02-10T05:44:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 107729 12.66 MB/sec execute 79 sec latency 42.488 ms 2024-02-10T05:44:26.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 108511 12.72 MB/sec execute 80 sec latency 38.758 ms 2024-02-10T05:44:27.103 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:26 smithi028 ceph-mon[24680]: pgmap v38: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.6 MiB/s wr, 463 op/s 2024-02-10T05:44:27.104 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:26 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:27.104 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:26 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:27.104 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:26 smithi028 ceph-mon[24680]: Reconfiguring prometheus.smithi028 (dependencies changed)... 2024-02-10T05:44:27.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 108999 12.71 MB/sec execute 81 sec latency 64.622 ms 2024-02-10T05:44:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:26 smithi192 ceph-mon[26904]: pgmap v38: 97 pgs: 97 active+clean; 329 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.6 MiB/s wr, 463 op/s 2024-02-10T05:44:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:26 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:26 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:26 smithi192 ceph-mon[26904]: Reconfiguring prometheus.smithi028 (dependencies changed)... 2024-02-10T05:44:28.116 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:27 smithi028 ceph-mon[24680]: Reconfiguring daemon prometheus.smithi028 on smithi028 2024-02-10T05:44:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 109619 12.75 MB/sec execute 82 sec latency 34.696 ms 2024-02-10T05:44:28.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:27 smithi192 ceph-mon[26904]: Reconfiguring daemon prometheus.smithi028 on smithi028 2024-02-10T05:44:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 110024 12.82 MB/sec execute 83 sec latency 38.026 ms 2024-02-10T05:44:29.123 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:28 smithi028 ceph-mon[24680]: pgmap v39: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 833 op/s 2024-02-10T05:44:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:28 smithi192 ceph-mon[26904]: pgmap v39: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 833 op/s 2024-02-10T05:44:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 110440 12.78 MB/sec execute 84 sec latency 38.172 ms 2024-02-10T05:44:31.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:30 smithi192 ceph-mon[26904]: pgmap v40: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.8 MiB/s wr, 581 op/s 2024-02-10T05:44:31.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 110816 12.76 MB/sec execute 85 sec latency 49.816 ms 2024-02-10T05:44:31.139 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:30 smithi028 ceph-mon[24680]: pgmap v40: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.8 MiB/s wr, 581 op/s 2024-02-10T05:44:32.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 111067 12.69 MB/sec execute 86 sec latency 91.984 ms 2024-02-10T05:44:32.155 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:31 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:32.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:31 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:33.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 111421 12.65 MB/sec execute 87 sec latency 38.744 ms 2024-02-10T05:44:33.162 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:32 smithi028 ceph-mon[24680]: pgmap v41: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.8 MiB/s wr, 581 op/s 2024-02-10T05:44:33.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:32 smithi192 ceph-mon[26904]: pgmap v41: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.8 MiB/s wr, 581 op/s 2024-02-10T05:44:34.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 111780 12.60 MB/sec execute 88 sec latency 64.054 ms 2024-02-10T05:44:34.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:34 smithi028 ceph-mon[24680]: pgmap v42: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 829 op/s 2024-02-10T05:44:34.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:34 smithi192 ceph-mon[26904]: pgmap v42: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 829 op/s 2024-02-10T05:44:35.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 112136 12.53 MB/sec execute 89 sec latency 37.798 ms 2024-02-10T05:44:36.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 112450 12.49 MB/sec execute 90 sec latency 40.160 ms 2024-02-10T05:44:36.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:36 smithi028 ceph-mon[24680]: pgmap v43: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.7 MiB/s wr, 618 op/s 2024-02-10T05:44:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 112910 12.48 MB/sec execute 91 sec latency 42.448 ms 2024-02-10T05:44:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:36 smithi192 ceph-mon[26904]: pgmap v43: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.7 MiB/s wr, 618 op/s 2024-02-10T05:44:38.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 113350 12.48 MB/sec execute 92 sec latency 35.030 ms 2024-02-10T05:44:39.002 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:38 smithi028 ceph-mon[24680]: pgmap v44: 97 pgs: 97 active+clean; 334 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 839 op/s 2024-02-10T05:44:39.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 113770 12.54 MB/sec execute 93 sec latency 41.889 ms 2024-02-10T05:44:39.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:38 smithi192 ceph-mon[26904]: pgmap v44: 97 pgs: 97 active+clean; 334 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 839 op/s 2024-02-10T05:44:40.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 114217 12.51 MB/sec execute 94 sec latency 32.467 ms 2024-02-10T05:44:41.033 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:40 smithi028 ceph-mon[24680]: pgmap v45: 97 pgs: 97 active+clean; 334 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.7 MiB/s wr, 469 op/s 2024-02-10T05:44:41.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:40 smithi192 ceph-mon[26904]: pgmap v45: 97 pgs: 97 active+clean; 334 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.7 MiB/s wr, 469 op/s 2024-02-10T05:44:41.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 114563 12.47 MB/sec execute 95 sec latency 33.094 ms 2024-02-10T05:44:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 114979 12.47 MB/sec execute 96 sec latency 36.568 ms 2024-02-10T05:44:43.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:42 smithi028 ceph-mon[24680]: pgmap v46: 97 pgs: 97 active+clean; 334 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.7 MiB/s wr, 469 op/s 2024-02-10T05:44:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 115641 12.47 MB/sec execute 97 sec latency 29.402 ms 2024-02-10T05:44:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:42 smithi192 ceph-mon[26904]: pgmap v46: 97 pgs: 97 active+clean; 334 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.7 MiB/s wr, 469 op/s 2024-02-10T05:44:44.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 116250 12.50 MB/sec execute 98 sec latency 34.454 ms 2024-02-10T05:44:44.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:44 smithi028 ceph-mon[24680]: pgmap v47: 97 pgs: 97 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 721 op/s 2024-02-10T05:44:44.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:44 smithi192 ceph-mon[26904]: pgmap v47: 97 pgs: 97 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 721 op/s 2024-02-10T05:44:45.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 116814 12.55 MB/sec execute 99 sec latency 85.028 ms 2024-02-10T05:44:46.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 117386 12.62 MB/sec execute 100 sec latency 29.774 ms 2024-02-10T05:44:47.083 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:46 smithi028 ceph-mon[24680]: pgmap v48: 97 pgs: 97 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 473 op/s 2024-02-10T05:44:47.083 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:46 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:47.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 117806 12.59 MB/sec execute 101 sec latency 33.620 ms 2024-02-10T05:44:47.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:46 smithi192 ceph-mon[26904]: pgmap v48: 97 pgs: 97 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.9 MiB/s wr, 473 op/s 2024-02-10T05:44:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:46 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:48.092 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:47 smithi028 ceph-mon[24680]: mgrmap e28: smithi192.hppzcm(active, since 92s), standbys: smithi028.ucuqfx 2024-02-10T05:44:48.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 118170 12.58 MB/sec execute 102 sec latency 49.998 ms 2024-02-10T05:44:48.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:47 smithi192 ceph-mon[26904]: mgrmap e28: smithi192.hppzcm(active, since 92s), standbys: smithi028.ucuqfx 2024-02-10T05:44:49.102 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:48 smithi028 ceph-mon[24680]: pgmap v49: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 808 op/s 2024-02-10T05:44:49.102 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:48 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:49.102 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:48 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:49.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 118631 12.55 MB/sec execute 103 sec latency 31.948 ms 2024-02-10T05:44:49.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:48 smithi192 ceph-mon[26904]: pgmap v49: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 808 op/s 2024-02-10T05:44:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:48 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:48 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:50.113 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:49 smithi028 ceph-mon[24680]: Reconfiguring node-exporter.smithi192 (dependencies changed)... 2024-02-10T05:44:50.114 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:49 smithi028 ceph-mon[24680]: Reconfiguring daemon node-exporter.smithi192 on smithi192 2024-02-10T05:44:50.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 119266 12.59 MB/sec execute 104 sec latency 28.264 ms 2024-02-10T05:44:50.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:49 smithi192 ceph-mon[26904]: Reconfiguring node-exporter.smithi192 (dependencies changed)... 2024-02-10T05:44:50.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:49 smithi192 ceph-mon[26904]: Reconfiguring daemon node-exporter.smithi192 on smithi192 2024-02-10T05:44:51.117 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:50 smithi028 ceph-mon[24680]: pgmap v50: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.3 MiB/s wr, 587 op/s 2024-02-10T05:44:51.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 119891 12.59 MB/sec execute 105 sec latency 31.930 ms 2024-02-10T05:44:51.175 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:50 smithi192 ceph-mon[26904]: pgmap v50: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.3 MiB/s wr, 587 op/s 2024-02-10T05:44:52.121 INFO:teuthology.orchestra.run.smithi028.stdout: 5 120361 12.59 MB/sec execute 106 sec latency 47.297 ms 2024-02-10T05:44:52.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:51 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:52.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:51 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:52.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:51 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:44:52.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:51 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T05:44:52.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:51 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:52.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:51 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:44:52.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:51 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi028.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T05:44:52.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:51 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:52.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:51 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:44:52.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:51 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi028.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T05:44:52.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:51 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:52.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:51 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:44:52.143 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:44:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:51 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:51 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:51 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:44:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:51 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T05:44:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:51 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:51 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:44:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:51 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi028.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T05:44:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:51 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:51 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:44:52.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:51 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi028.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T05:44:52.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:51 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:52.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:51 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:44:53.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 120907 12.68 MB/sec execute 107 sec latency 27.303 ms 2024-02-10T05:44:53.166 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:52 smithi028 ceph-mon[24680]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:44:53.167 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:52 smithi028 ceph-mon[24680]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T05:44:53.167 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:52 smithi028 ceph-mon[24680]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:44:53.167 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:52 smithi028 ceph-mon[24680]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi028.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T05:44:53.167 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:52 smithi028 ceph-mon[24680]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:44:53.167 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:52 smithi028 ceph-mon[24680]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi028.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T05:44:53.168 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:52 smithi028 ceph-mon[24680]: Upgrade: Need to upgrade myself (mgr.smithi192.hppzcm) 2024-02-10T05:44:53.168 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:52 smithi028 ceph-mon[24680]: Upgrade: Need to upgrade myself (mgr.smithi192.hppzcm) 2024-02-10T05:44:53.168 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:52 smithi028 ceph-mon[24680]: pgmap v51: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.3 MiB/s wr, 587 op/s 2024-02-10T05:44:53.195 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:44:53.195 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 starting - - - - 2024-02-10T05:44:53.195 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (10m) 83s ago 10m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:44:53.195 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (9m) 82s ago 9m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:44:53.195 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 starting - - - - 2024-02-10T05:44:53.195 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (4m) 83s ago 4m 36.5M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:44:53.195 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (4m) 82s ago 4m 13.0M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:44:53.195 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:9283 running (11m) 83s ago 11m 333M - 17.2.0 e1d6a67b021e dc390c86451c 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283 running (108s) 82s ago 9m 488M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddfed28689ae 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (11m) 83s ago 11m 49.3M 2048M 17.2.0 e1d6a67b021e a5294cba5afa 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (9m) 82s ago 9m 40.4M 2048M 17.2.0 e1d6a67b021e cf2f3de335d9 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (4m) 83s ago 3m 262M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (3m) 82s ago 3m 73.3M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 starting - - - - 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 starting - - - - 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (8m) 83s ago 8m 326M 1937M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (8m) 83s ago 8m 255M 1937M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (7m) 83s ago 7m 207M 1937M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (7m) 83s ago 7m 267M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (6m) 82s ago 6m 308M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (6m) 82s ago 6m 321M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:44:53.196 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (6m) 82s ago 6m 264M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:44:53.197 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (5m) 82s ago 5m 396M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:44:53.197 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 starting - - - - 2024-02-10T05:44:53.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:52 smithi192 ceph-mon[26904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:44:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:52 smithi192 ceph-mon[26904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T05:44:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:52 smithi192 ceph-mon[26904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:44:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:52 smithi192 ceph-mon[26904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi028.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T05:44:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:52 smithi192 ceph-mon[26904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:44:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:52 smithi192 ceph-mon[26904]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi028.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T05:44:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:52 smithi192 ceph-mon[26904]: Upgrade: Need to upgrade myself (mgr.smithi192.hppzcm) 2024-02-10T05:44:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:52 smithi192 ceph-mon[26904]: Upgrade: Need to upgrade myself (mgr.smithi192.hppzcm) 2024-02-10T05:44:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:52 smithi192 ceph-mon[26904]: pgmap v51: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.3 MiB/s wr, 587 op/s 2024-02-10T05:44:53.734 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:44:53.734 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:44:53.734 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:44:53.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:44:53.735 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:44:54.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 121225 12.65 MB/sec execute 108 sec latency 49.661 ms 2024-02-10T05:44:54.178 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:54 smithi028 ceph-mon[24680]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:54.178 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:54 smithi028 ceph-mon[24680]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:54.178 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:54 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:54.178 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:54 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi028.ucuqfx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:44:54.178 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:54 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi028.ucuqfx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:44:54.178 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:54 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:44:54.178 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:54 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:44:54.179 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:54 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/928019312' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:44:54.208 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:44:54.208 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:44:54.209 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:44:54.209 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:44:54.209 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-02-10T05:44:54.209 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "1/23 daemons upgraded", 2024-02-10T05:44:54.209 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading mgr daemons", 2024-02-10T05:44:54.209 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:44:54.209 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:44:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:54 smithi192 ceph-mon[26904]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:54 smithi192 ceph-mon[26904]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:54 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:44:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:54 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi028.ucuqfx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:44:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:54 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi028.ucuqfx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:44:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:54 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:44:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:54 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:44:54.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:54 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/928019312' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:44:54.720 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:44:55.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 121542 12.60 MB/sec execute 109 sec latency 38.014 ms 2024-02-10T05:44:55.197 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:55 smithi028 ceph-mon[24680]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:55.197 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:55 smithi028 ceph-mon[24680]: Upgrade: Updating mgr.smithi028.ucuqfx 2024-02-10T05:44:55.197 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:55 smithi028 ceph-mon[24680]: Deploying daemon mgr.smithi028.ucuqfx on smithi028 2024-02-10T05:44:55.197 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:55 smithi028 ceph-mon[24680]: pgmap v52: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 902 op/s 2024-02-10T05:44:55.197 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:55 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2683964802' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:44:55.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:55 smithi192 ceph-mon[26904]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:55.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:55 smithi192 ceph-mon[26904]: Upgrade: Updating mgr.smithi028.ucuqfx 2024-02-10T05:44:55.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:55 smithi192 ceph-mon[26904]: Deploying daemon mgr.smithi028.ucuqfx on smithi028 2024-02-10T05:44:55.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:55 smithi192 ceph-mon[26904]: pgmap v52: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 902 op/s 2024-02-10T05:44:55.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:55 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2683964802' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:44:56.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 121930 12.57 MB/sec execute 110 sec latency 29.951 ms 2024-02-10T05:44:56.206 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:56 smithi028 ceph-mon[24680]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:56.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:56 smithi192 ceph-mon[26904]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:57.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 122396 12.57 MB/sec execute 111 sec latency 34.592 ms 2024-02-10T05:44:57.211 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:57 smithi028 ceph-mon[24680]: pgmap v53: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.5 MiB/s wr, 650 op/s 2024-02-10T05:44:57.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:57 smithi192 ceph-mon[26904]: pgmap v53: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.5 MiB/s wr, 650 op/s 2024-02-10T05:44:58.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 122889 12.55 MB/sec execute 112 sec latency 33.358 ms 2024-02-10T05:44:59.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 123286 12.52 MB/sec execute 113 sec latency 37.824 ms 2024-02-10T05:44:59.164 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:44:59 smithi028 ceph-mon[24680]: pgmap v54: 97 pgs: 97 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 891 op/s 2024-02-10T05:44:59.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:44:59 smithi192 ceph-mon[26904]: pgmap v54: 97 pgs: 97 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 891 op/s 2024-02-10T05:45:00.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 123746 12.52 MB/sec execute 114 sec latency 34.813 ms 2024-02-10T05:45:00.947 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:00 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:00.948 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:00 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:00.948 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:00 smithi028 ceph-mon[24680]: pgmap v55: 97 pgs: 97 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.6 MiB/s wr, 556 op/s 2024-02-10T05:45:00.948 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:00 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:45:00.948 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:00 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:00.948 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:00 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:01.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 124130 12.56 MB/sec execute 115 sec latency 32.763 ms 2024-02-10T05:45:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:00 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:00 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:00 smithi192 ceph-mon[26904]: pgmap v55: 97 pgs: 97 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.6 MiB/s wr, 556 op/s 2024-02-10T05:45:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:00 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:45:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:00 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:00 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:01.956 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:01 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:45:02.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:01 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:45:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 124600 12.55 MB/sec execute 116 sec latency 36.716 ms 2024-02-10T05:45:02.965 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:02 smithi028 ceph-mon[24680]: pgmap v56: 97 pgs: 97 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.6 MiB/s wr, 556 op/s 2024-02-10T05:45:03.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 125123 12.56 MB/sec execute 117 sec latency 31.043 ms 2024-02-10T05:45:03.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:02 smithi192 ceph-mon[26904]: pgmap v56: 97 pgs: 97 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.6 MiB/s wr, 556 op/s 2024-02-10T05:45:04.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 125437 12.53 MB/sec execute 118 sec latency 49.099 ms 2024-02-10T05:45:04.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:04 smithi192 ceph-mon[26904]: pgmap v57: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 821 op/s 2024-02-10T05:45:04.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:04 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:04.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:04 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:04.505 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:04 smithi028 ceph-mon[24680]: pgmap v57: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 821 op/s 2024-02-10T05:45:04.506 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:04 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:04.506 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:04 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:05.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 125779 12.52 MB/sec execute 119 sec latency 47.719 ms 2024-02-10T05:45:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 126112 12.45 MB/sec execute 120 sec latency 67.094 ms 2024-02-10T05:45:07.049 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:06 smithi028 ceph-mon[24680]: pgmap v58: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 506 op/s 2024-02-10T05:45:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 126430 12.41 MB/sec execute 121 sec latency 36.130 ms 2024-02-10T05:45:07.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:06 smithi192 ceph-mon[26904]: pgmap v58: 97 pgs: 97 active+clean; 342 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 506 op/s 2024-02-10T05:45:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 126666 12.33 MB/sec execute 122 sec latency 50.837 ms 2024-02-10T05:45:09.092 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:08 smithi028 ceph-mon[24680]: pgmap v59: 97 pgs: 97 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 744 op/s 2024-02-10T05:45:09.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 127017 12.34 MB/sec execute 123 sec latency 63.230 ms 2024-02-10T05:45:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:08 smithi192 ceph-mon[26904]: pgmap v59: 97 pgs: 97 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 744 op/s 2024-02-10T05:45:10.118 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:09 smithi028 ceph-mon[24680]: Standby manager daemon smithi028.ucuqfx restarted 2024-02-10T05:45:10.119 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:09 smithi028 ceph-mon[24680]: Standby manager daemon smithi028.ucuqfx started 2024-02-10T05:45:10.119 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:09 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi028.ucuqfx/crt"}]: dispatch 2024-02-10T05:45:10.119 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:09 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:45:10.119 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:09 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi028.ucuqfx/key"}]: dispatch 2024-02-10T05:45:10.119 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:09 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:45:10.119 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:09 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:10.119 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:09 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:10.122 INFO:teuthology.orchestra.run.smithi028.stdout: 5 127320 12.29 MB/sec execute 124 sec latency 37.192 ms 2024-02-10T05:45:10.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:09 smithi192 ceph-mon[26904]: Standby manager daemon smithi028.ucuqfx restarted 2024-02-10T05:45:10.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:09 smithi192 ceph-mon[26904]: Standby manager daemon smithi028.ucuqfx started 2024-02-10T05:45:10.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:09 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi028.ucuqfx/crt"}]: dispatch 2024-02-10T05:45:10.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:09 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:45:10.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:09 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi028.ucuqfx/key"}]: dispatch 2024-02-10T05:45:10.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:09 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:45:10.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:09 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:10.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:09 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:11.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 127699 12.32 MB/sec execute 125 sec latency 45.639 ms 2024-02-10T05:45:11.157 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:10 smithi028 ceph-mon[24680]: pgmap v60: 97 pgs: 97 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 503 op/s 2024-02-10T05:45:11.158 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:10 smithi028 ceph-mon[24680]: mgrmap e29: smithi192.hppzcm(active, since 115s), standbys: smithi028.ucuqfx 2024-02-10T05:45:11.158 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:10 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:11.158 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:10 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:10 smithi192 ceph-mon[26904]: pgmap v60: 97 pgs: 97 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 503 op/s 2024-02-10T05:45:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:10 smithi192 ceph-mon[26904]: mgrmap e29: smithi192.hppzcm(active, since 115s), standbys: smithi028.ucuqfx 2024-02-10T05:45:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:10 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:10 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:12.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 128071 12.30 MB/sec execute 126 sec latency 46.870 ms 2024-02-10T05:45:13.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 128415 12.28 MB/sec execute 127 sec latency 36.358 ms 2024-02-10T05:45:13.192 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:12 smithi028 ceph-mon[24680]: pgmap v61: 97 pgs: 97 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 503 op/s 2024-02-10T05:45:13.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:12 smithi192 ceph-mon[26904]: pgmap v61: 97 pgs: 97 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 503 op/s 2024-02-10T05:45:14.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 128816 12.27 MB/sec execute 128 sec latency 50.743 ms 2024-02-10T05:45:14.460 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:14 smithi028 ceph-mon[24680]: pgmap v62: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 696 op/s 2024-02-10T05:45:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:14 smithi192 ceph-mon[26904]: pgmap v62: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 696 op/s 2024-02-10T05:45:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 129315 12.29 MB/sec execute 129 sec latency 42.770 ms 2024-02-10T05:45:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 129728 12.23 MB/sec execute 130 sec latency 41.982 ms 2024-02-10T05:45:17.009 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:16 smithi028 ceph-mon[24680]: pgmap v63: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 431 op/s 2024-02-10T05:45:17.010 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:45:17.010 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:16 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:17.010 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:16 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:17.010 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:45:17.010 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:45:17.010 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:16 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:17.010 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:16 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:17.010 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:45:17.010 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:16 smithi028 ceph-mon[24680]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mgr fail", "who": "smithi192.hppzcm"}]: dispatch 2024-02-10T05:45:17.010 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:16 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mgr fail", "who": "smithi192.hppzcm"}]: dispatch 2024-02-10T05:45:17.010 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:16 smithi028 ceph-mon[24680]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T05:45:17.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 130030 12.19 MB/sec execute 131 sec latency 63.391 ms 2024-02-10T05:45:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:16 smithi192 ceph-mon[26904]: pgmap v63: 97 pgs: 97 active+clean; 342 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 431 op/s 2024-02-10T05:45:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:45:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:16 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:16 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:45:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:45:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:16 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:16 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' 2024-02-10T05:45:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:45:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:16 smithi192 ceph-mon[26904]: from='mgr.24425 172.21.15.192:0/3705650865' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mgr fail", "who": "smithi192.hppzcm"}]: dispatch 2024-02-10T05:45:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:16 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "mgr fail", "who": "smithi192.hppzcm"}]: dispatch 2024-02-10T05:45:17.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:16 smithi192 ceph-mon[26904]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T05:45:18.024 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: Upgrade: Need to upgrade myself (mgr.smithi192.hppzcm) 2024-02-10T05:45:18.024 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: Upgrade: Need to upgrade myself (mgr.smithi192.hppzcm) 2024-02-10T05:45:18.024 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: Failing over to other MGR 2024-02-10T05:45:18.024 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd='[{"prefix": "mgr fail", "who": "smithi192.hppzcm"}]': finished 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: mgrmap e30: smithi028.ucuqfx(active, starting, since 0.85827s) 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi028.xhtowe"}]: dispatch 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi192.hdcbzw"}]: dispatch 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi028.ucuqfx", "id": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:45:18.025 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:45:18.026 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:45:18.026 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:45:18.026 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:45:18.026 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:45:18.026 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: Manager daemon smithi028.ucuqfx is now available 2024-02-10T05:45:18.026 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:45:18.026 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:45:18.026 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:45:18.026 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:45:18.026 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:17 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:45:18.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 130311 12.14 MB/sec execute 132 sec latency 51.597 ms 2024-02-10T05:45:18.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: Upgrade: Need to upgrade myself (mgr.smithi192.hppzcm) 2024-02-10T05:45:18.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: Upgrade: Need to upgrade myself (mgr.smithi192.hppzcm) 2024-02-10T05:45:18.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: Failing over to other MGR 2024-02-10T05:45:18.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.24425 ' entity='mgr.smithi192.hppzcm' cmd='[{"prefix": "mgr fail", "who": "smithi192.hppzcm"}]': finished 2024-02-10T05:45:18.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: mgrmap e30: smithi028.ucuqfx(active, starting, since 0.85827s) 2024-02-10T05:45:18.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:45:18.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:45:18.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi028.xhtowe"}]: dispatch 2024-02-10T05:45:18.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi192.hdcbzw"}]: dispatch 2024-02-10T05:45:18.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi028.ucuqfx", "id": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:45:18.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: Manager daemon smithi028.ucuqfx is now available 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:45:18.307 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:45:18.308 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:17 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:45:19.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 130600 12.13 MB/sec execute 133 sec latency 74.897 ms 2024-02-10T05:45:19.596 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:19 smithi028 ceph-mon[24680]: mgrmap e31: smithi028.ucuqfx(active, since 1.86283s) 2024-02-10T05:45:19.597 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:19 smithi028 ceph-mon[24680]: pgmap v3: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-10T05:45:19.597 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:19 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:19.597 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:19 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:19.723 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:19 smithi192 ceph-mon[26904]: mgrmap e31: smithi028.ucuqfx(active, since 1.86283s) 2024-02-10T05:45:19.723 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:19 smithi192 ceph-mon[26904]: pgmap v3: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-10T05:45:19.723 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:19 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:19.723 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:19 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:20.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 130902 12.09 MB/sec execute 134 sec latency 40.805 ms 2024-02-10T05:45:20.598 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:20 smithi192 ceph-mon[26904]: [10/Feb/2024:05:45:18] ENGINE Bus STARTING 2024-02-10T05:45:20.598 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:20 smithi192 ceph-mon[26904]: [10/Feb/2024:05:45:18] ENGINE Serving on http://172.21.15.28:8765 2024-02-10T05:45:20.598 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:20 smithi192 ceph-mon[26904]: [10/Feb/2024:05:45:19] ENGINE Serving on https://172.21.15.28:7150 2024-02-10T05:45:20.599 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:20 smithi192 ceph-mon[26904]: [10/Feb/2024:05:45:19] ENGINE Bus STARTED 2024-02-10T05:45:20.599 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:20 smithi192 ceph-mon[26904]: pgmap v4: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-10T05:45:20.599 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:20 smithi192 ceph-mon[26904]: mgrmap e32: smithi028.ucuqfx(active, since 3s) 2024-02-10T05:45:20.613 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:20 smithi028 ceph-mon[24680]: [10/Feb/2024:05:45:18] ENGINE Bus STARTING 2024-02-10T05:45:20.613 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:20 smithi028 ceph-mon[24680]: [10/Feb/2024:05:45:18] ENGINE Serving on http://172.21.15.28:8765 2024-02-10T05:45:20.613 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:20 smithi028 ceph-mon[24680]: [10/Feb/2024:05:45:19] ENGINE Serving on https://172.21.15.28:7150 2024-02-10T05:45:20.613 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:20 smithi028 ceph-mon[24680]: [10/Feb/2024:05:45:19] ENGINE Bus STARTED 2024-02-10T05:45:20.614 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:20 smithi028 ceph-mon[24680]: pgmap v4: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-10T05:45:20.614 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:20 smithi028 ceph-mon[24680]: mgrmap e32: smithi028.ucuqfx(active, since 3s) 2024-02-10T05:45:21.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 131308 12.13 MB/sec execute 135 sec latency 32.250 ms 2024-02-10T05:45:22.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 131660 12.10 MB/sec execute 136 sec latency 60.319 ms 2024-02-10T05:45:22.638 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:22 smithi028 ceph-mon[24680]: pgmap v5: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-10T05:45:22.639 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:22 smithi028 ceph-mon[24680]: mgrmap e33: smithi028.ucuqfx(active, since 4s) 2024-02-10T05:45:22.815 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:22 smithi192 ceph-mon[26904]: pgmap v5: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-10T05:45:22.815 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:22 smithi192 ceph-mon[26904]: mgrmap e33: smithi028.ucuqfx(active, since 4s) 2024-02-10T05:45:23.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 132008 12.08 MB/sec execute 137 sec latency 46.832 ms 2024-02-10T05:45:24.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 132323 12.07 MB/sec execute 138 sec latency 42.025 ms 2024-02-10T05:45:24.156 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:24 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:24.157 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:24 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:24.157 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:24 smithi028 ceph-mon[24680]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 528 KiB/s wr, 20 op/s 2024-02-10T05:45:24.241 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:24 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:24.241 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:24 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:24.241 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:24 smithi192 ceph-mon[26904]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 528 KiB/s wr, 20 op/s 2024-02-10T05:45:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 132660 12.03 MB/sec execute 139 sec latency 50.732 ms 2024-02-10T05:45:25.255 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:45:25.679 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:25 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:25.679 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:25 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:25.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:25 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:25.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:25 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:26.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 133005 12.02 MB/sec execute 140 sec latency 79.732 ms 2024-02-10T05:45:26.190 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:45:26.190 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (94s) 16s ago 10m 17.5M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:45:26.190 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (10m) 16s ago 10m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:45:26.190 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (10m) 3s ago 10m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:45:26.190 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (65s) 16s ago 10m 44.6M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:45:26.190 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (4m) 16s ago 4m 36.8M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:45:26.190 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (4m) 3s ago 4m 13.6M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:45:26.190 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (27s) 16s ago 11m 93.9M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:45:26.190 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283 running (2m) 3s ago 10m 99.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ddfed28689ae 2024-02-10T05:45:26.190 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (12m) 16s ago 12m 52.4M 2048M 17.2.0 e1d6a67b021e a5294cba5afa 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (10m) 3s ago 10m 43.9M 2048M 17.2.0 e1d6a67b021e cf2f3de335d9 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (4m) 16s ago 4m 273M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (4m) 3s ago 4m 79.2M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (61s) 16s ago 10m 9571k - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (35s) 3s ago 9m 11.3M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (9m) 16s ago 9m 461M 1937M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (8m) 16s ago 8m 399M 1937M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (8m) 16s ago 8m 300M 1937M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (7m) 16s ago 7m 407M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (7m) 3s ago 7m 461M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (7m) 3s ago 7m 494M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (6m) 3s ago 6m 406M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (6m) 3s ago 6m 568M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:45:26.191 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (39s) 16s ago 9m 47.9M - 2.33.4 514e6a882f6e 71be72780ab9 2024-02-10T05:45:26.698 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:45:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:45:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:45:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:45:26.698 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:45:26.699 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:45:26.721 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:26 smithi028 ceph-mon[24680]: from='client.14836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:45:26.722 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:26 smithi028 ceph-mon[24680]: pgmap v7: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 34 KiB/s rd, 4.1 MiB/s wr, 362 op/s 2024-02-10T05:45:26.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:26 smithi192 ceph-mon[26904]: from='client.14836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:45:26.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:26 smithi192 ceph-mon[26904]: pgmap v7: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 34 KiB/s rd, 4.1 MiB/s wr, 362 op/s 2024-02-10T05:45:27.109 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:45:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:45:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:45:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:45:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:45:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr" 2024-02-10T05:45:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:45:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "2/23 daemons upgraded", 2024-02-10T05:45:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-02-10T05:45:27.110 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:45:27.110 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:45:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 133298 11.96 MB/sec execute 141 sec latency 71.442 ms 2024-02-10T05:45:27.720 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:45:27.745 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:27 smithi028 ceph-mon[24680]: from='client.24553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:45:27.746 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:27 smithi028 ceph-mon[24680]: from='client.14844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:45:27.746 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:27 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3253364076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:45:27.746 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:27 smithi028 ceph-mon[24680]: Standby manager daemon smithi192.hppzcm started 2024-02-10T05:45:27.746 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:27 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/2627437517' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/crt"}]: dispatch 2024-02-10T05:45:27.746 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:27 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/2627437517' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:45:27.746 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:27 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/2627437517' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/key"}]: dispatch 2024-02-10T05:45:27.746 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:27 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/2627437517' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:45:27.746 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:27 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:27.746 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:27 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:27 smithi192 ceph-mon[26904]: from='client.24553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:45:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:27 smithi192 ceph-mon[26904]: from='client.14844 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:45:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:27 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/3253364076' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:45:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:27 smithi192 ceph-mon[26904]: Standby manager daemon smithi192.hppzcm started 2024-02-10T05:45:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:27 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/2627437517' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/crt"}]: dispatch 2024-02-10T05:45:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:27 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/2627437517' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:45:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:27 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/2627437517' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/key"}]: dispatch 2024-02-10T05:45:27.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:27 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/2627437517' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:45:27.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:27 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:27.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:27 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 133550 11.92 MB/sec execute 142 sec latency 64.686 ms 2024-02-10T05:45:28.792 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:28 smithi028 ceph-mon[24680]: from='client.14852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:45:28.792 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:28 smithi028 ceph-mon[24680]: mgrmap e34: smithi028.ucuqfx(active, since 10s), standbys: smithi192.hppzcm 2024-02-10T05:45:28.792 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:28 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi192.hppzcm", "id": "smithi192.hppzcm"}]: dispatch 2024-02-10T05:45:28.792 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:28 smithi028 ceph-mon[24680]: pgmap v8: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 3.2 MiB/s wr, 281 op/s 2024-02-10T05:45:28.792 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:28 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/3967065930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:45:28.792 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:28 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:28.792 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:28 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:28 smithi192 ceph-mon[26904]: from='client.14852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:45:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:28 smithi192 ceph-mon[26904]: mgrmap e34: smithi028.ucuqfx(active, since 10s), standbys: smithi192.hppzcm 2024-02-10T05:45:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:28 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi192.hppzcm", "id": "smithi192.hppzcm"}]: dispatch 2024-02-10T05:45:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:28 smithi192 ceph-mon[26904]: pgmap v8: 97 pgs: 97 active+clean; 344 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 3.2 MiB/s wr, 281 op/s 2024-02-10T05:45:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:28 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/3967065930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:45:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:28 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:28.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:28 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: 5 133807 11.86 MB/sec execute 143 sec latency 47.230 ms 2024-02-10T05:45:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 134154 11.87 MB/sec execute 144 sec latency 62.159 ms 2024-02-10T05:45:30.661 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:30 smithi028 ceph-mon[24680]: pgmap v9: 97 pgs: 97 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.5 MiB/s wr, 447 op/s 2024-02-10T05:45:31.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:30 smithi192 ceph-mon[26904]: pgmap v9: 97 pgs: 97 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.5 MiB/s wr, 447 op/s 2024-02-10T05:45:31.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 134796 11.94 MB/sec execute 145 sec latency 21.239 ms 2024-02-10T05:45:32.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 135293 11.96 MB/sec execute 146 sec latency 40.347 ms 2024-02-10T05:45:32.677 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:32 smithi028 ceph-mon[24680]: pgmap v10: 97 pgs: 97 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.1 MiB/s wr, 408 op/s 2024-02-10T05:45:33.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:32 smithi192 ceph-mon[26904]: pgmap v10: 97 pgs: 97 active+clean; 338 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.1 MiB/s wr, 408 op/s 2024-02-10T05:45:33.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 135666 11.92 MB/sec execute 147 sec latency 50.824 ms 2024-02-10T05:45:33.705 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:33 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:45:33.977 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:33 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:45:34.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 136060 11.92 MB/sec execute 148 sec latency 36.978 ms 2024-02-10T05:45:34.763 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:34 smithi028 ceph-mon[24680]: pgmap v11: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.8 MiB/s wr, 436 op/s 2024-02-10T05:45:34.764 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:34 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:34.764 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:34 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:34.764 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:34 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:45:34.764 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:34 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:45:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:34 smithi192 ceph-mon[26904]: pgmap v11: 97 pgs: 97 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.8 MiB/s wr, 436 op/s 2024-02-10T05:45:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:34 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:34 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:34 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:45:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:34 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:45:35.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 136347 11.91 MB/sec execute 149 sec latency 41.923 ms 2024-02-10T05:45:35.739 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:35 smithi028 ceph-mon[24680]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-10T05:45:35.739 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:35 smithi028 ceph-mon[24680]: Updating smithi192:/etc/ceph/ceph.conf 2024-02-10T05:45:35.739 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:35 smithi028 ceph-mon[24680]: Updating smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:45:35.739 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:35 smithi028 ceph-mon[24680]: Updating smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:45:35.739 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:35 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:35.740 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:35 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:35.740 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:35 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:35.740 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:35 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:35.740 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:35 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:35.740 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:35 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:36.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:35 smithi192 ceph-mon[26904]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-10T05:45:36.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:35 smithi192 ceph-mon[26904]: Updating smithi192:/etc/ceph/ceph.conf 2024-02-10T05:45:36.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:35 smithi192 ceph-mon[26904]: Updating smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:45:36.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:35 smithi192 ceph-mon[26904]: Updating smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:45:36.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:35 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:36.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:35 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:36.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:35 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:36.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:35 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:36.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:35 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:36.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:35 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:36.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 136636 11.87 MB/sec execute 150 sec latency 48.653 ms 2024-02-10T05:45:36.797 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:36 smithi028 ceph-mon[24680]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:45:36.798 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:36 smithi028 ceph-mon[24680]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:45:36.798 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:36 smithi028 ceph-mon[24680]: Updating smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:45:36.798 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:36 smithi028 ceph-mon[24680]: Updating smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:45:36.798 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:36 smithi028 ceph-mon[24680]: pgmap v12: 97 pgs: 97 active+clean; 347 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 6.7 MiB/s wr, 655 op/s 2024-02-10T05:45:36.798 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:36 smithi028 ceph-mon[24680]: Reconfiguring prometheus.smithi028 (dependencies changed)... 2024-02-10T05:45:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:36 smithi192 ceph-mon[26904]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:45:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:36 smithi192 ceph-mon[26904]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:45:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:36 smithi192 ceph-mon[26904]: Updating smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:45:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:36 smithi192 ceph-mon[26904]: Updating smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:45:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:36 smithi192 ceph-mon[26904]: pgmap v12: 97 pgs: 97 active+clean; 347 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 6.7 MiB/s wr, 655 op/s 2024-02-10T05:45:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:36 smithi192 ceph-mon[26904]: Reconfiguring prometheus.smithi028 (dependencies changed)... 2024-02-10T05:45:37.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 136991 11.83 MB/sec execute 151 sec latency 49.572 ms 2024-02-10T05:45:37.824 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:37 smithi028 ceph-mon[24680]: Reconfiguring daemon prometheus.smithi028 on smithi028 2024-02-10T05:45:38.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:37 smithi192 ceph-mon[26904]: Reconfiguring daemon prometheus.smithi028 on smithi028 2024-02-10T05:45:38.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 137260 11.78 MB/sec execute 152 sec latency 47.338 ms 2024-02-10T05:45:38.834 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:38 smithi028 ceph-mon[24680]: pgmap v13: 97 pgs: 97 active+clean; 347 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 453 op/s 2024-02-10T05:45:39.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:38 smithi192 ceph-mon[26904]: pgmap v13: 97 pgs: 97 active+clean; 347 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 453 op/s 2024-02-10T05:45:39.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 137900 11.83 MB/sec execute 153 sec latency 34.429 ms 2024-02-10T05:45:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: 5 138586 11.90 MB/sec execute 154 sec latency 26.211 ms 2024-02-10T05:45:40.843 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:40 smithi028 ceph-mon[24680]: pgmap v14: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 658 op/s 2024-02-10T05:45:41.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:40 smithi192 ceph-mon[26904]: pgmap v14: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 658 op/s 2024-02-10T05:45:41.127 INFO:teuthology.orchestra.run.smithi028.stdout: 5 139259 11.93 MB/sec execute 155 sec latency 65.045 ms 2024-02-10T05:45:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 140020 12.01 MB/sec execute 156 sec latency 33.554 ms 2024-02-10T05:45:42.854 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:42 smithi028 ceph-mon[24680]: pgmap v15: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-02-10T05:45:43.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:42 smithi192 ceph-mon[26904]: pgmap v15: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-02-10T05:45:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: 5 140660 12.00 MB/sec execute 157 sec latency 34.285 ms 2024-02-10T05:45:44.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 141336 12.05 MB/sec execute 158 sec latency 35.985 ms 2024-02-10T05:45:44.359 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:44 smithi028 ceph-mon[24680]: pgmap v16: 97 pgs: 97 active+clean; 342 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 554 op/s 2024-02-10T05:45:44.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:44 smithi192 ceph-mon[26904]: pgmap v16: 97 pgs: 97 active+clean; 342 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 554 op/s 2024-02-10T05:45:45.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 142085 12.12 MB/sec execute 159 sec latency 36.094 ms 2024-02-10T05:45:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:45 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:45 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:45 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:45:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:45 smithi192 ceph-mon[26904]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:45:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:45 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:45:46.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:45 smithi192 ceph-mon[26904]: pgmap v17: 97 pgs: 97 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 8.2 MiB/s wr, 833 op/s 2024-02-10T05:45:46.123 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:45 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:46.124 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:45 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:46.124 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:45 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:45:46.124 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:45 smithi028 ceph-mon[24680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:45:46.124 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:45 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:45:46.124 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:45 smithi028 ceph-mon[24680]: pgmap v17: 97 pgs: 97 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 8.2 MiB/s wr, 833 op/s 2024-02-10T05:45:46.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 142709 12.14 MB/sec execute 160 sec latency 33.696 ms 2024-02-10T05:45:47.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 143447 12.22 MB/sec execute 161 sec latency 19.904 ms 2024-02-10T05:45:47.379 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:47 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.hppzcm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:45:47.380 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:47 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:45:47.380 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:47 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:45:47.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:47 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.hppzcm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:45:47.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:47 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:45:47.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:47 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:45:48.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 144026 12.20 MB/sec execute 162 sec latency 44.424 ms 2024-02-10T05:45:48.392 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:48 smithi028 ceph-mon[24680]: Upgrade: Updating mgr.smithi192.hppzcm 2024-02-10T05:45:48.393 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:48 smithi028 ceph-mon[24680]: Deploying daemon mgr.smithi192.hppzcm on smithi192 2024-02-10T05:45:48.393 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:48 smithi028 ceph-mon[24680]: pgmap v18: 97 pgs: 97 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.0 MiB/s wr, 605 op/s 2024-02-10T05:45:48.393 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:48 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:45:48.493 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:48 smithi192 ceph-mon[26904]: Upgrade: Updating mgr.smithi192.hppzcm 2024-02-10T05:45:48.493 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:48 smithi192 ceph-mon[26904]: Deploying daemon mgr.smithi192.hppzcm on smithi192 2024-02-10T05:45:48.493 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:48 smithi192 ceph-mon[26904]: pgmap v18: 97 pgs: 97 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.0 MiB/s wr, 605 op/s 2024-02-10T05:45:48.493 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:48 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:45:49.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 144467 12.16 MB/sec execute 163 sec latency 56.964 ms 2024-02-10T05:45:50.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 145096 12.21 MB/sec execute 164 sec latency 35.360 ms 2024-02-10T05:45:50.668 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:50 smithi028 ceph-mon[24680]: pgmap v19: 97 pgs: 97 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 976 op/s 2024-02-10T05:45:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:50 smithi192 ceph-mon[26904]: pgmap v19: 97 pgs: 97 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 976 op/s 2024-02-10T05:45:51.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 145828 12.29 MB/sec execute 165 sec latency 29.184 ms 2024-02-10T05:45:52.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 146289 12.28 MB/sec execute 166 sec latency 39.958 ms 2024-02-10T05:45:52.784 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:52 smithi028 ceph-mon[24680]: pgmap v20: 97 pgs: 97 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 772 op/s 2024-02-10T05:45:52.784 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:52 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:52.785 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:52 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:52.785 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:52 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:45:53.017 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:52 smithi192 ceph-mon[26904]: pgmap v20: 97 pgs: 97 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 772 op/s 2024-02-10T05:45:53.017 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:52 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:53.017 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:52 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:53.017 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:52 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:45:53.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 146590 12.27 MB/sec execute 167 sec latency 39.607 ms 2024-02-10T05:45:54.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 146812 12.23 MB/sec execute 168 sec latency 65.236 ms 2024-02-10T05:45:54.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:54 smithi028 ceph-mon[24680]: pgmap v21: 97 pgs: 97 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 15 KiB/s rd, 8.2 MiB/s wr, 774 op/s 2024-02-10T05:45:54.465 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:54 smithi192 ceph-mon[26904]: pgmap v21: 97 pgs: 97 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 15 KiB/s rd, 8.2 MiB/s wr, 774 op/s 2024-02-10T05:45:55.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 147087 12.21 MB/sec execute 169 sec latency 66.147 ms 2024-02-10T05:45:56.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 147626 12.19 MB/sec execute 170 sec latency 34.194 ms 2024-02-10T05:45:56.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:56 smithi028 ceph-mon[24680]: pgmap v22: 97 pgs: 97 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 11 MiB/s wr, 969 op/s 2024-02-10T05:45:56.904 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:56 smithi192 ceph-mon[26904]: pgmap v22: 97 pgs: 97 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 11 MiB/s wr, 969 op/s 2024-02-10T05:45:57.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 148086 12.17 MB/sec execute 171 sec latency 77.427 ms 2024-02-10T05:45:58.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 148548 12.19 MB/sec execute 172 sec latency 30.673 ms 2024-02-10T05:45:58.319 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:45:58.399 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:58 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:58.400 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:58 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:58.400 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:58 smithi028 ceph-mon[24680]: pgmap v23: 97 pgs: 97 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 662 op/s 2024-02-10T05:45:58.547 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:58 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:58.547 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:58 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:58.547 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:58 smithi192 ceph-mon[26904]: pgmap v23: 97 pgs: 97 active+clean; 352 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.2 MiB/s wr, 662 op/s 2024-02-10T05:45:59.129 INFO:teuthology.orchestra.run.smithi028.stdout: 5 149044 12.22 MB/sec execute 173 sec latency 34.950 ms 2024-02-10T05:45:59.461 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:45:59.461 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (2m) 32s ago 11m 17.5M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:45:59.461 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (11m) 32s ago 11m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:45:59.461 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (10m) 2s ago 10m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:45:59.461 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (98s) 32s ago 11m 44.7M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:45:59.461 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (5m) 32s ago 5m 37.1M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:45:59.461 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (5m) 2s ago 5m 13.9M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:45:59.461 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (60s) 32s ago 12m 477M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:45:59.461 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (8s) 2s ago 10m 66.5M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:45:59.461 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (12m) 32s ago 12m 54.0M 2048M 17.2.0 e1d6a67b021e a5294cba5afa 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (10m) 2s ago 10m 43.2M 2048M 17.2.0 e1d6a67b021e cf2f3de335d9 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (5m) 32s ago 5m 277M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (5m) 2s ago 5m 80.7M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (95s) 32s ago 11m 9893k - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (68s) 2s ago 10m 14.9M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (9m) 32s ago 9m 479M 1937M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (9m) 32s ago 9m 403M 1937M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (8m) 32s ago 8m 309M 1937M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (8m) 32s ago 8m 427M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (8m) 2s ago 8m 492M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (7m) 2s ago 7m 558M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (7m) 2s ago 7m 468M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (6m) 2s ago 6m 351M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:45:59.462 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 starting - - - - 2024-02-10T05:45:59.655 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:59 smithi028 ceph-mon[24680]: from='client.24577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:45:59.655 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:59 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:45:59.656 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:45:59 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:00.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:59 smithi192 ceph-mon[26904]: from='client.24577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:00.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:59 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:00.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:45:59 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:00.106 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:46:00.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:46:00.106 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:46:00.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:46:00.107 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:46:00.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 149419 12.21 MB/sec execute 174 sec latency 31.459 ms 2024-02-10T05:46:00.615 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:46:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:46:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:46:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:46:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:46:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr" 2024-02-10T05:46:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:46:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "2/23 daemons upgraded", 2024-02-10T05:46:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading mgr daemons", 2024-02-10T05:46:00.616 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:46:00.616 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:46:00.667 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:00 smithi028 ceph-mon[24680]: from='client.24581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:00.668 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:00 smithi028 ceph-mon[24680]: from='client.24585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:00.668 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:00 smithi028 ceph-mon[24680]: pgmap v24: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 9.6 MiB/s wr, 916 op/s 2024-02-10T05:46:00.668 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:00 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/1701870348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:46:00.955 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:00 smithi192 ceph-mon[26904]: from='client.24581 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:00.955 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:00 smithi192 ceph-mon[26904]: from='client.24585 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:00.955 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:00 smithi192 ceph-mon[26904]: pgmap v24: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 9.6 MiB/s wr, 916 op/s 2024-02-10T05:46:00.955 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:00 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/1701870348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:46:01.128 INFO:teuthology.orchestra.run.smithi028.stdout: 5 149737 12.18 MB/sec execute 175 sec latency 41.766 ms 2024-02-10T05:46:01.315 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:46:01.697 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:01 smithi028 ceph-mon[24680]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:01.697 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:01 smithi028 ceph-mon[24680]: Standby manager daemon smithi192.hppzcm restarted 2024-02-10T05:46:01.697 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:01 smithi028 ceph-mon[24680]: Standby manager daemon smithi192.hppzcm started 2024-02-10T05:46:01.697 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:01 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/1810067605' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/crt"}]: dispatch 2024-02-10T05:46:01.697 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:01 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/1810067605' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:46:01.697 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:01 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/1810067605' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/key"}]: dispatch 2024-02-10T05:46:01.697 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:01 smithi028 ceph-mon[24680]: from='mgr.? 172.21.15.192:0/1810067605' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:46:01.697 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:01 smithi028 ceph-mon[24680]: from='client.? 172.21.15.28:0/2275607113' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:46:02.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:01 smithi192 ceph-mon[26904]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:02.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:01 smithi192 ceph-mon[26904]: Standby manager daemon smithi192.hppzcm restarted 2024-02-10T05:46:02.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:01 smithi192 ceph-mon[26904]: Standby manager daemon smithi192.hppzcm started 2024-02-10T05:46:02.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:01 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/1810067605' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/crt"}]: dispatch 2024-02-10T05:46:02.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:01 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/1810067605' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:46:02.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:01 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/1810067605' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/key"}]: dispatch 2024-02-10T05:46:02.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:01 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/1810067605' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:46:02.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:01 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/2275607113' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:46:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: 5 150029 12.16 MB/sec execute 176 sec latency 80.112 ms 2024-02-10T05:46:02.739 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:02 smithi028 ceph-mon[24680]: pgmap v25: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.6 MiB/s wr, 545 op/s 2024-02-10T05:46:02.739 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:02 smithi028 ceph-mon[24680]: mgrmap e35: smithi028.ucuqfx(active, since 44s), standbys: smithi192.hppzcm 2024-02-10T05:46:03.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:02 smithi192 ceph-mon[26904]: pgmap v25: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.6 MiB/s wr, 545 op/s 2024-02-10T05:46:03.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:02 smithi192 ceph-mon[26904]: mgrmap e35: smithi028.ucuqfx(active, since 44s), standbys: smithi192.hppzcm 2024-02-10T05:46:03.129 INFO:teuthology.orchestra.run.smithi028.stdout: 5 150311 12.14 MB/sec execute 177 sec latency 78.684 ms 2024-02-10T05:46:03.805 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:46:03.805 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:03.805 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:03.805 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:03.805 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:03.805 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:46:03.805 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:46:03.805 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:03.805 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:03.805 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:46:03.806 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:46:03.806 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:03.806 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi028.ucuqfx"}]: dispatch 2024-02-10T05:46:03.806 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi028.ucuqfx"}]': finished 2024-02-10T05:46:03.806 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi192.hppzcm"}]: dispatch 2024-02-10T05:46:03.806 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi192.hppzcm"}]': finished 2024-02-10T05:46:03.806 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:03 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T05:46:04.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:46:04.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:04.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:04.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:04.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:04.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:46:04.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:46:04.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:04.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:04.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:46:04.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:46:04.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:04.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi028.ucuqfx"}]: dispatch 2024-02-10T05:46:04.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi028.ucuqfx"}]': finished 2024-02-10T05:46:04.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi192.hppzcm"}]: dispatch 2024-02-10T05:46:04.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi192.hppzcm"}]': finished 2024-02-10T05:46:04.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:03 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T05:46:04.129 INFO:teuthology.orchestra.run.smithi028.stdout: 5 150585 12.13 MB/sec execute 178 sec latency 76.013 ms 2024-02-10T05:46:04.747 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:04 smithi028 ceph-mon[24680]: Upgrade: Setting container_image for all mgr 2024-02-10T05:46:04.748 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:04 smithi028 ceph-mon[24680]: pgmap v26: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 5.8 MiB/s wr, 600 op/s 2024-02-10T05:46:04.821 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:04 smithi192 ceph-mon[26904]: Upgrade: Setting container_image for all mgr 2024-02-10T05:46:04.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:04 smithi192 ceph-mon[26904]: pgmap v26: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 5.8 MiB/s wr, 600 op/s 2024-02-10T05:46:05.132 INFO:teuthology.orchestra.run.smithi028.stdout: 5 150956 12.09 MB/sec execute 179 sec latency 51.059 ms 2024-02-10T05:46:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: 5 151297 12.05 MB/sec execute 180 sec latency 53.048 ms 2024-02-10T05:46:06.869 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:06 smithi028 ceph-mon[24680]: pgmap v27: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 7.9 MiB/s wr, 748 op/s 2024-02-10T05:46:06.870 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:06 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:06.870 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:06 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:46:06.870 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:06 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:46:06.870 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:06 smithi028 ceph-mon[24680]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:46:07.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:06 smithi192 ceph-mon[26904]: pgmap v27: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 7.9 MiB/s wr, 748 op/s 2024-02-10T05:46:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:06 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:06 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:46:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:06 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:46:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:06 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:46:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: 5 151672 12.04 MB/sec execute 181 sec latency 42.333 ms 2024-02-10T05:46:07.884 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:07 smithi028 ceph-mon[24680]: Upgrade: Updating mon.smithi028 2024-02-10T05:46:07.884 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:07 smithi028 ceph-mon[24680]: Deploying daemon mon.smithi028 on smithi028 2024-02-10T05:46:08.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:07 smithi192 ceph-mon[26904]: Upgrade: Updating mon.smithi028 2024-02-10T05:46:08.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:07 smithi192 ceph-mon[26904]: Deploying daemon mon.smithi028 on smithi028 2024-02-10T05:46:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: 5 152240 12.09 MB/sec execute 182 sec latency 33.369 ms 2024-02-10T05:46:08.904 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:08 smithi028 ceph-mon[24680]: pgmap v28: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.6 MiB/s wr, 458 op/s 2024-02-10T05:46:09.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:08 smithi192 ceph-mon[26904]: pgmap v28: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.6 MiB/s wr, 458 op/s 2024-02-10T05:46:09.132 INFO:teuthology.orchestra.run.smithi028.stdout: 5 152781 12.10 MB/sec execute 183 sec latency 41.947 ms 2024-02-10T05:46:09.413 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:09 smithi028 systemd[1]: Stopping Ceph mon.smithi028 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:46:09.924 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:09 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028[24676]: 2024-02-10T05:46:09.740+0000 7f72401c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi028 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:46:09.924 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:09 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028[24676]: 2024-02-10T05:46:09.740+0000 7f72401c0700 -1 mon.smithi028@0(leader) e2 *** Got Signal Terminated *** 2024-02-10T05:46:09.924 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:09 smithi028 podman[66171]: 2024-02-10 05:46:09.773815807 +0000 UTC m=+0.186393646 container died a5294cba5afaeca41b2bd8d72367849733d1306c18d2d8b9bde7ece6804f6ad8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.buildah.version=1.19.8, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , name=centos-stream, version=8, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-02-10T05:46:10.132 INFO:teuthology.orchestra.run.smithi028.stdout: 5 153164 12.10 MB/sec execute 184 sec latency 47.409 ms 2024-02-10T05:46:10.686 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:10 smithi028 podman[66171]: 2024-02-10 05:46:10.544454898 +0000 UTC m=+0.957032729 container cleanup a5294cba5afaeca41b2bd8d72367849733d1306c18d2d8b9bde7ece6804f6ad8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, vendor=Red Hat, Inc., distribution-scope=public, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, release=754, ceph=True, io.buildah.version=1.19.8, GIT_CLEAN=True, name=centos-stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container) 2024-02-10T05:46:10.686 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:10 smithi028 bash[66171]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028 2024-02-10T05:46:11.139 INFO:teuthology.orchestra.run.smithi028.stdout: 5 153695 12.11 MB/sec execute 185 sec latency 38.183 ms 2024-02-10T05:46:11.193 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:10 smithi028 podman[66192]: 2024-02-10 05:46:10.989054462 +0000 UTC m=+1.210595550 container remove a5294cba5afaeca41b2bd8d72367849733d1306c18d2d8b9bde7ece6804f6ad8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., version=8, release=754, distribution-scope=public, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD) 2024-02-10T05:46:11.444 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:11 smithi028 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi028.service: Deactivated successfully. 2024-02-10T05:46:11.445 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:11 smithi028 systemd[1]: Stopped Ceph mon.smithi028 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:46:11.445 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:11 smithi028 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi028.service: Consumed 16.992s CPU time. 2024-02-10T05:46:11.956 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:11 smithi028 systemd[1]: Starting Ceph mon.smithi028 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:46:12.144 INFO:teuthology.orchestra.run.smithi028.stdout: 5 154052 12.11 MB/sec execute 186 sec latency 35.247 ms 2024-02-10T05:46:12.207 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 podman[66320]: 2024-02-10 05:46:11.94530863 +0000 UTC m=+0.027757727 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:46:12.207 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 podman[66320]: 2024-02-10 05:46:12.191868227 +0000 UTC m=+0.274317364 container create 57f180a5eb092445eed580f154079b0915072f1725a7107427e3c401a16b5a4a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-10T05:46:12.712 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 podman[66320]: 2024-02-10 05:46:12.699335678 +0000 UTC m=+0.781784782 container init 57f180a5eb092445eed580f154079b0915072f1725a7107427e3c401a16b5a4a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:46:12.713 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 podman[66320]: 2024-02-10 05:46:12.706524508 +0000 UTC m=+0.788973604 container start 57f180a5eb092445eed580f154079b0915072f1725a7107427e3c401a16b5a4a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-10T05:46:12.965 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: pidfile_write: ignore empty --pid-file 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: load: jerasure load: lrc 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: RocksDB version: 7.9.2 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Git sha 0 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: DB SUMMARY 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: DB Session ID: 763J44JSXLF3UVA9SHT4 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: MANIFEST file: MANIFEST-000015 size: 978 Bytes 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi028/store.db dir, Total Num: 1, files: 000030.sst 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi028/store.db: 000028.log size: 2941770 ; 2024-02-10T05:46:12.966 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.env: 0x55e9c269cae0 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.info_log: 0x55e9c4533760 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.statistics: (nil) 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.use_fsync: 0 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:46:12.967 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:46:12.968 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:46:12.968 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:46:12.968 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:46:12.968 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:46:12.968 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:46:12.968 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:46:12.968 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:46:12.968 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.db_log_dir: 2024-02-10T05:46:12.968 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.wal_dir: 2024-02-10T05:46:12.968 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:46:12.969 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:46:12.969 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:46:12.969 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:46:12.969 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:46:12.969 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.write_buffer_manager: 0x55e9c4507400 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.unordered_write: 0 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:46:12.970 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.row_cache: None 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.wal_filter: None 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.wal_compression: 0 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:46:12.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:46:12.972 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:46:12.972 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:46:12.972 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:46:12.972 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:46:12.972 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:46:12.972 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:46:12.972 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:46:12.972 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T05:46:12.972 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:46:12.972 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:46:12.972 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_open_files: -1 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:46:12.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Compression algorithms supported: 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: kZSTD supported: 0 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi028/store.db/MANIFEST-000015 2024-02-10T05:46:12.974 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.merge_operator: 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_filter: None 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e9c4533880) 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: index_type: 0 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: data_block_index_type: 0 2024-02-10T05:46:12.975 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: index_shortening: 1 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: checksum: 4 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: no_block_cache: 0 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_cache: 0x55e9c4537090 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_cache_options: 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: capacity : 536870912 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: num_shard_bits : 4 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: strict_capacity_limit : 0 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_cache_compressed: (nil) 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: persistent_cache: (nil) 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_size: 4096 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_size_deviation: 10 2024-02-10T05:46:12.976 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_restart_interval: 16 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: index_block_restart_interval: 1 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: metadata_block_size: 4096 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: partition_filters: 0 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: use_delta_encoding: 1 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: filter_policy: bloomfilter 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: whole_key_filtering: 1 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: verify_compression: 0 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: format_version: 5 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: enable_index_compression: 1 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: block_align: 0 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: max_auto_readahead_size: 262144 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: prepopulate_block_cache: 0 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: initial_auto_readahead_size: 8192 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T05:46:12.977 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compression: NoCompression 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.num_levels: 7 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:46:12.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:46:12.979 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:46:12.979 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:46:12.979 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:46:12.979 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:46:12.979 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:46:12.979 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:46:12.979 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:46:12.980 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:46:12.981 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:46:12.982 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:46:12.983 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:46:12.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:46:12.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:46:12.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:46:12.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.ttl: 2592000 2024-02-10T05:46:12.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:46:12.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T05:46:12.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T05:46:12.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:46:12.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:46:12.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi028/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11791, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5119927e-0a17-4cd0-b3e2-50f5153f9c1b 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543972747210, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-02-10T05:46:12.985 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543972758433, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2666926, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11792, "largest_seqno": 13124, "table_properties": {"data_size": 2660458, "index_size": 4041, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1413, "raw_key_size": 12480, "raw_average_key_size": 22, "raw_value_size": 2648324, "raw_average_value_size": 4806, "num_data_blocks": 185, "num_entries": 551, "num_filter_entries": 551, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707543972, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5119927e-0a17-4cd0-b3e2-50f5153f9c1b", "db_session_id": "763J44JSXLF3UVA9SHT4", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543972758631, "job": 1, "event": "recovery_finished"} 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi028/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e9c4626000 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: DB pointer 0x55e9c4612000 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ** DB Stats ** 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:46:12.986 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ** Compaction Stats [default] ** 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: L0 1/0 2.54 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 426.1 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: L6 1/0 6.28 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Sum 2/0 8.82 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 426.1 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 426.1 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ** Compaction Stats [default] ** 2024-02-10T05:46:12.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 426.1 0.01 0.00 1 0.006 0 0 0.0 0.0 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Cumulative compaction: 0.00 GB write, 177.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Interval compaction: 0.00 GB write, 177.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:46:12.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:46:12.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Block cache BinnedLRUCache@0x55e9c4537090#2 capacity: 512.00 MB usage: 5.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-02-10T05:46:12.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.53 KB,0.000292063%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-02-10T05:46:12.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:46:12.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:46:12.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: starting mon.smithi028 rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi028 fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:46:12.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: mon.smithi028@-1(???) e2 preinit fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:46:12.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: mon.smithi028@-1(???).mds e7 new map 2024-02-10T05:46:12.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: mon.smithi028@-1(???).mds e7 print_map 2024-02-10T05:46:12.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: e7 2024-02-10T05:46:12.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:46:12.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T05:46:12.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: legacy client fscid: 1 2024-02-10T05:46:12.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:46:12.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Filesystem 'foofs' (1) 2024-02-10T05:46:12.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: fs_name foofs 2024-02-10T05:46:12.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: epoch 5 2024-02-10T05:46:12.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-10T05:46:12.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: created 2024-02-10T05:40:28.932045+0000 2024-02-10T05:46:12.991 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: modified 2024-02-10T05:40:35.175134+0000 2024-02-10T05:46:12.991 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: tableserver 0 2024-02-10T05:46:12.991 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: root 0 2024-02-10T05:46:12.991 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: session_timeout 60 2024-02-10T05:46:12.991 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: session_autoclose 300 2024-02-10T05:46:12.991 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: max_file_size 1099511627776 2024-02-10T05:46:12.991 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: max_xattr_size 65536 2024-02-10T05:46:12.992 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: required_client_features {} 2024-02-10T05:46:12.992 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: last_failure 0 2024-02-10T05:46:12.992 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: last_failure_osd_epoch 0 2024-02-10T05:46:12.992 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T05:46:12.992 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: max_mds 1 2024-02-10T05:46:12.992 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: in 0 2024-02-10T05:46:12.992 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: up {0=14482} 2024-02-10T05:46:12.992 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: failed 2024-02-10T05:46:12.992 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: damaged 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: stopped 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: data_pools [3] 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: metadata_pool 2 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: inline_data disabled 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: balancer 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: bal_rank_mask -1 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: standby_count_wanted 1 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: [mds.foofs.smithi028.xhtowe{0:14482} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.28:6834/350665837,v1:172.21.15.28:6835/350665837] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: Standby daemons: 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout: [mds.foofs.smithi192.hdcbzw{-1:24275} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6832/853070905,v1:172.21.15.192:6833/853070905] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: mon.smithi028@-1(???).osd e73 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-10T05:46:12.993 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: mon.smithi028@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T05:46:12.994 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: mon.smithi028@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T05:46:12.994 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: mon.smithi028@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T05:46:12.994 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 ceph-mon[66336]: mon.smithi028@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-02-10T05:46:12.994 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 bash[66320]: 57f180a5eb092445eed580f154079b0915072f1725a7107427e3c401a16b5a4a 2024-02-10T05:46:12.994 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:12 smithi028 systemd[1]: Started Ceph mon.smithi028 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:46:13.150 INFO:teuthology.orchestra.run.smithi028.stdout: 5 154419 12.07 MB/sec execute 187 sec latency 53.098 ms 2024-02-10T05:46:14.151 INFO:teuthology.orchestra.run.smithi028.stdout: 5 154861 12.05 MB/sec execute 188 sec latency 37.743 ms 2024-02-10T05:46:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: 5 155202 12.03 MB/sec execute 189 sec latency 483.826 ms 2024-02-10T05:46:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: 5 155202 11.97 MB/sec execute 190 sec latency 1483.868 ms 2024-02-10T05:46:17.151 INFO:teuthology.orchestra.run.smithi028.stdout: 5 155202 11.91 MB/sec execute 191 sec latency 2483.926 ms 2024-02-10T05:46:18.151 INFO:teuthology.orchestra.run.smithi028.stdout: 5 155202 11.84 MB/sec execute 192 sec latency 3483.983 ms 2024-02-10T05:46:19.151 INFO:teuthology.orchestra.run.smithi028.stdout: 5 155216 11.78 MB/sec execute 193 sec latency 4465.153 ms 2024-02-10T05:46:19.384 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:46:19.384 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:46:19.384 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: pgmap v30: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 451 op/s 2024-02-10T05:46:19.384 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: pgmap v31: 97 pgs: 97 active+clean; 346 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.2 MiB/s wr, 593 op/s 2024-02-10T05:46:19.384 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: pgmap v32: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.9 MiB/s wr, 651 op/s 2024-02-10T05:46:19.384 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: pgmap v33: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.9 MiB/s wr, 500 op/s 2024-02-10T05:46:19.385 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:46:19.385 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: mon.smithi028 calling monitor election 2024-02-10T05:46:19.385 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: mon.smithi028 is new leader, mons smithi028,smithi192 in quorum (ranks 0,1) 2024-02-10T05:46:19.385 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: monmap e2: 2 mons at {smithi028=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:46:19.385 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 1 up:standby 2024-02-10T05:46:19.385 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T05:46:19.385 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: mgrmap e35: smithi028.ucuqfx(active, since 62s), standbys: smithi192.hppzcm 2024-02-10T05:46:19.385 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: mgrmap e36: smithi028.ucuqfx(active, since 62s), standbys: smithi192.hppzcm 2024-02-10T05:46:19.385 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: overall HEALTH_OK 2024-02-10T05:46:19.385 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:19 smithi028 ceph-mon[66336]: from='mgr.14806 ' entity='' 2024-02-10T05:46:19.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:46:19.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:46:19.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: pgmap v30: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 451 op/s 2024-02-10T05:46:19.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: pgmap v31: 97 pgs: 97 active+clean; 346 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.2 MiB/s wr, 593 op/s 2024-02-10T05:46:19.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: pgmap v32: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.9 MiB/s wr, 651 op/s 2024-02-10T05:46:19.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: pgmap v33: 97 pgs: 97 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.9 MiB/s wr, 500 op/s 2024-02-10T05:46:19.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: from='mgr.14806 172.21.15.28:0/1821545925' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:46:19.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: mon.smithi028 calling monitor election 2024-02-10T05:46:19.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: mon.smithi028 is new leader, mons smithi028,smithi192 in quorum (ranks 0,1) 2024-02-10T05:46:19.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: monmap e2: 2 mons at {smithi028=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:46:19.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 1 up:standby 2024-02-10T05:46:19.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T05:46:19.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: mgrmap e35: smithi028.ucuqfx(active, since 62s), standbys: smithi192.hppzcm 2024-02-10T05:46:19.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: mgrmap e36: smithi028.ucuqfx(active, since 62s), standbys: smithi192.hppzcm 2024-02-10T05:46:19.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: overall HEALTH_OK 2024-02-10T05:46:19.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:19 smithi192 ceph-mon[26904]: from='mgr.14806 ' entity='' 2024-02-10T05:46:20.151 INFO:teuthology.orchestra.run.smithi028.stdout: 5 155934 11.86 MB/sec execute 194 sec latency 16.886 ms 2024-02-10T05:46:21.151 INFO:teuthology.orchestra.run.smithi028.stdout: 5 156453 11.87 MB/sec execute 195 sec latency 30.524 ms 2024-02-10T05:46:21.398 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:21 smithi028 ceph-mon[66336]: mgrmap e37: smithi028.ucuqfx(active, since 63s), standbys: smithi192.hppzcm 2024-02-10T05:46:21.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:21 smithi192 ceph-mon[26904]: mgrmap e37: smithi028.ucuqfx(active, since 63s), standbys: smithi192.hppzcm 2024-02-10T05:46:22.151 INFO:teuthology.orchestra.run.smithi028.stdout: 5 156773 11.85 MB/sec execute 196 sec latency 40.452 ms 2024-02-10T05:46:23.151 INFO:teuthology.orchestra.run.smithi028.stdout: 5 157131 11.84 MB/sec execute 197 sec latency 38.974 ms 2024-02-10T05:46:24.151 INFO:teuthology.orchestra.run.smithi028.stdout: 5 157647 11.87 MB/sec execute 198 sec latency 41.004 ms 2024-02-10T05:46:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: 5 158157 11.85 MB/sec execute 199 sec latency 42.020 ms 2024-02-10T05:46:26.152 INFO:teuthology.orchestra.run.smithi028.stdout: 5 158602 11.82 MB/sec execute 200 sec latency 37.826 ms 2024-02-10T05:46:27.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 159171 11.86 MB/sec execute 201 sec latency 26.196 ms 2024-02-10T05:46:27.848 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:27 smithi028 ceph-mon[66336]: Standby manager daemon smithi192.hppzcm restarted 2024-02-10T05:46:27.849 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:27 smithi028 ceph-mon[66336]: Standby manager daemon smithi192.hppzcm started 2024-02-10T05:46:27.849 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:27 smithi028 ceph-mon[66336]: from='mgr.? 172.21.15.192:0/1586757175' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/crt"}]: dispatch 2024-02-10T05:46:27.849 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:27 smithi028 ceph-mon[66336]: from='mgr.? 172.21.15.192:0/1586757175' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:46:27.849 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:27 smithi028 ceph-mon[66336]: from='mgr.? 172.21.15.192:0/1586757175' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/key"}]: dispatch 2024-02-10T05:46:27.849 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:27 smithi028 ceph-mon[66336]: from='mgr.? 172.21.15.192:0/1586757175' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:46:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:27 smithi192 ceph-mon[26904]: Standby manager daemon smithi192.hppzcm restarted 2024-02-10T05:46:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:27 smithi192 ceph-mon[26904]: Standby manager daemon smithi192.hppzcm started 2024-02-10T05:46:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:27 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/1586757175' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/crt"}]: dispatch 2024-02-10T05:46:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:27 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/1586757175' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:46:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:27 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/1586757175' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi192.hppzcm/key"}]: dispatch 2024-02-10T05:46:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:27 smithi192 ceph-mon[26904]: from='mgr.? 172.21.15.192:0/1586757175' entity='mgr.smithi192.hppzcm' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:46:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 159776 11.89 MB/sec execute 202 sec latency 52.974 ms 2024-02-10T05:46:28.866 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:28 smithi028 ceph-mon[66336]: mgrmap e38: smithi028.ucuqfx(active, since 71s), standbys: smithi192.hppzcm 2024-02-10T05:46:29.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:28 smithi192 ceph-mon[26904]: mgrmap e38: smithi028.ucuqfx(active, since 71s), standbys: smithi192.hppzcm 2024-02-10T05:46:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 160146 11.89 MB/sec execute 203 sec latency 29.759 ms 2024-02-10T05:46:30.129 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: Active manager daemon smithi028.ucuqfx restarted 2024-02-10T05:46:30.129 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: Activating manager daemon smithi028.ucuqfx 2024-02-10T05:46:30.129 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T05:46:30.129 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: mgrmap e39: smithi028.ucuqfx(active, starting, since 0.00841165s), standbys: smithi192.hppzcm 2024-02-10T05:46:30.129 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi028.xhtowe"}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi192.hdcbzw"}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi028.ucuqfx", "id": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi192.hppzcm", "id": "smithi192.hppzcm"}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:46:30.130 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:46:30.131 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:46:30.131 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:46:30.131 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:29 smithi028 ceph-mon[66336]: Manager daemon smithi028.ucuqfx is now available 2024-02-10T05:46:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 160511 11.88 MB/sec execute 204 sec latency 44.909 ms 2024-02-10T05:46:30.163 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192[26899]: 2024-02-10T05:46:29.908+0000 7ff1a5d45700 -1 mon.smithi192@1(peon).osd e74 update_from_paxos full map CRC mismatch, resetting to canonical 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: Active manager daemon smithi028.ucuqfx restarted 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: Activating manager daemon smithi028.ucuqfx 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: mgrmap e39: smithi028.ucuqfx(active, starting, since 0.00841165s), standbys: smithi192.hppzcm 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi028.xhtowe"}]: dispatch 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi192.hdcbzw"}]: dispatch 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi028.ucuqfx", "id": "smithi028.ucuqfx"}]: dispatch 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr metadata", "who": "smithi192.hppzcm", "id": "smithi192.hppzcm"}]: dispatch 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:46:30.164 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:46:30.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:46:30.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:46:30.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:46:30.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:46:30.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:46:30.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:46:30.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:46:30.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:46:30.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:29 smithi192 ceph-mon[26904]: Manager daemon smithi028.ucuqfx is now available 2024-02-10T05:46:31.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:30 smithi028 ceph-mon[66336]: failed to encode map e74 with expected crc 2024-02-10T05:46:31.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:30 smithi028 ceph-mon[66336]: failed to encode map e74 with expected crc 2024-02-10T05:46:31.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:30 smithi028 ceph-mon[66336]: failed to encode map e74 with expected crc 2024-02-10T05:46:31.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:46:31.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:46:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:46:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:46:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:30 smithi028 ceph-mon[66336]: failed to encode map e74 with expected crc 2024-02-10T05:46:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:46:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:30 smithi028 ceph-mon[66336]: mgrmap e40: smithi028.ucuqfx(active, since 1.01142s), standbys: smithi192.hppzcm 2024-02-10T05:46:31.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 160842 11.87 MB/sec execute 205 sec latency 57.665 ms 2024-02-10T05:46:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:30 smithi192 ceph-mon[26904]: failed to encode map e74 with expected crc 2024-02-10T05:46:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:30 smithi192 ceph-mon[26904]: failed to encode map e74 with expected crc 2024-02-10T05:46:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:30 smithi192 ceph-mon[26904]: failed to encode map e74 with expected crc 2024-02-10T05:46:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:30 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:46:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:30 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:46:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:30 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:46:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:30 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:46:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:30 smithi192 ceph-mon[26904]: failed to encode map e74 with expected crc 2024-02-10T05:46:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:30 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi028.ucuqfx/trash_purge_schedule"}]: dispatch 2024-02-10T05:46:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:30 smithi192 ceph-mon[26904]: mgrmap e40: smithi028.ucuqfx(active, since 1.01142s), standbys: smithi192.hppzcm 2024-02-10T05:46:31.919 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:46:32.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 161064 11.84 MB/sec execute 206 sec latency 60.356 ms 2024-02-10T05:46:32.159 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:31 smithi028 ceph-mon[66336]: pgmap v3: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-10T05:46:32.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:31 smithi192 ceph-mon[26904]: pgmap v3: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-10T05:46:33.004 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:46:33.004 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (2m) 29s ago 11m 17.6M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:46:33.004 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (11m) 29s ago 11m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:46:33.004 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (11m) 35s ago 11m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:46:33.004 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (2m) 29s ago 11m 45.5M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (6m) 29s ago 5m 37.4M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (5m) 35s ago 5m 13.9M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (94s) 29s ago 13m 496M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (41s) 35s ago 11m 66.5M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (13m) 29s ago 13m 54.6M 2048M 17.2.0 e1d6a67b021e a5294cba5afa 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (11m) 35s ago 11m 43.2M 2048M 17.2.0 e1d6a67b021e cf2f3de335d9 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (5m) 29s ago 5m 280M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (5m) 35s ago 5m 80.7M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (2m) 29s ago 11m 12.2M - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (102s) 35s ago 10m 14.9M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (10m) 29s ago 10m 509M 1937M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (9m) 29s ago 9m 447M 1937M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:46:33.005 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (9m) 29s ago 9m 353M 1937M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:46:33.006 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (9m) 29s ago 9m 468M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:46:33.006 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (8m) 35s ago 8m 492M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:46:33.006 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (8m) 35s ago 8m 558M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:46:33.006 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (7m) 35s ago 7m 468M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:46:33.006 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (7m) 35s ago 7m 351M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:46:33.006 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (48s) 29s ago 10m 50.1M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:46:33.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 161325 11.83 MB/sec execute 207 sec latency 58.835 ms 2024-02-10T05:46:33.174 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:32 smithi028 ceph-mon[66336]: [10/Feb/2024:05:46:31] ENGINE Bus STARTING 2024-02-10T05:46:33.175 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:32 smithi028 ceph-mon[66336]: pgmap v4: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-10T05:46:33.175 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:32 smithi028 ceph-mon[66336]: [10/Feb/2024:05:46:32] ENGINE Serving on https://172.21.15.28:7150 2024-02-10T05:46:33.175 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:32 smithi028 ceph-mon[66336]: [10/Feb/2024:05:46:32] ENGINE Serving on http://172.21.15.28:8765 2024-02-10T05:46:33.175 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:32 smithi028 ceph-mon[66336]: [10/Feb/2024:05:46:32] ENGINE Bus STARTED 2024-02-10T05:46:33.175 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:32 smithi028 ceph-mon[66336]: mgrmap e41: smithi028.ucuqfx(active, since 3s), standbys: smithi192.hppzcm 2024-02-10T05:46:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:32 smithi192 ceph-mon[26904]: [10/Feb/2024:05:46:31] ENGINE Bus STARTING 2024-02-10T05:46:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:32 smithi192 ceph-mon[26904]: pgmap v4: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-10T05:46:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:32 smithi192 ceph-mon[26904]: [10/Feb/2024:05:46:32] ENGINE Serving on https://172.21.15.28:7150 2024-02-10T05:46:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:32 smithi192 ceph-mon[26904]: [10/Feb/2024:05:46:32] ENGINE Serving on http://172.21.15.28:8765 2024-02-10T05:46:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:32 smithi192 ceph-mon[26904]: [10/Feb/2024:05:46:32] ENGINE Bus STARTED 2024-02-10T05:46:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:32 smithi192 ceph-mon[26904]: mgrmap e41: smithi028.ucuqfx(active, since 3s), standbys: smithi192.hppzcm 2024-02-10T05:46:33.639 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:46:33.640 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:46:33.640 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T05:46:33.640 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T05:46:33.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:46:33.640 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:46:33.640 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:46:33.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:46:33.640 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:46:33.640 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:46:33.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:46:33.640 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:46:33.641 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:46:33.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:46:33.641 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:46:33.641 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-02-10T05:46:33.641 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T05:46:33.641 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:46:33.641 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:46:34.120 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:46:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:46:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:46:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:46:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:46:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr" 2024-02-10T05:46:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:46:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "2/23 daemons upgraded", 2024-02-10T05:46:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-02-10T05:46:34.120 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:46:34.120 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:46:34.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 161629 11.79 MB/sec execute 208 sec latency 58.150 ms 2024-02-10T05:46:34.216 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:33 smithi028 ceph-mon[66336]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:34.216 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:33 smithi028 ceph-mon[66336]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:34.217 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:33 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/3953128971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:46:34.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:33 smithi192 ceph-mon[26904]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:34.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:33 smithi192 ceph-mon[26904]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:34.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:33 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/3953128971' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:46:34.648 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:46:35.033 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:34 smithi028 ceph-mon[66336]: pgmap v5: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-10T05:46:35.033 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:34 smithi028 ceph-mon[66336]: mgrmap e42: smithi028.ucuqfx(active, since 4s), standbys: smithi192.hppzcm 2024-02-10T05:46:35.033 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:34 smithi028 ceph-mon[66336]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:35.033 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:34 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/1089139135' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:46:35.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 161964 11.75 MB/sec execute 209 sec latency 51.751 ms 2024-02-10T05:46:35.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:34 smithi192 ceph-mon[26904]: pgmap v5: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-10T05:46:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:34 smithi192 ceph-mon[26904]: mgrmap e42: smithi028.ucuqfx(active, since 4s), standbys: smithi192.hppzcm 2024-02-10T05:46:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:34 smithi192 ceph-mon[26904]: from='client.34140 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:46:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:34 smithi192 ceph-mon[26904]: from='client.? 172.21.15.28:0/1089139135' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:46:36.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 162201 11.72 MB/sec execute 210 sec latency 62.358 ms 2024-02-10T05:46:36.270 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:36 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:36.270 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:36 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:36.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:36 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:36.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:36 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:37.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 162451 11.71 MB/sec execute 211 sec latency 55.130 ms 2024-02-10T05:46:37.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:37 smithi028 ceph-mon[66336]: pgmap v6: 97 pgs: 97 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 41 KiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-02-10T05:46:37.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:37 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:37.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:37 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:37 smithi192 ceph-mon[26904]: pgmap v6: 97 pgs: 97 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 41 KiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-02-10T05:46:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:37 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:37 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:38.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 162803 11.73 MB/sec execute 212 sec latency 38.272 ms 2024-02-10T05:46:38.826 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:38 smithi028 ceph-mon[66336]: pgmap v7: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 2.6 MiB/s wr, 326 op/s 2024-02-10T05:46:39.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:38 smithi192 ceph-mon[26904]: pgmap v7: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 2.6 MiB/s wr, 326 op/s 2024-02-10T05:46:39.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 163329 11.73 MB/sec execute 213 sec latency 37.531 ms 2024-02-10T05:46:40.155 INFO:teuthology.orchestra.run.smithi028.stdout: 5 163648 11.72 MB/sec execute 214 sec latency 43.067 ms 2024-02-10T05:46:40.614 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:40 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:40.615 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:40 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:40.615 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:40 smithi028 ceph-mon[66336]: pgmap v8: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.0 MiB/s wr, 253 op/s 2024-02-10T05:46:40.615 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:40 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:40.615 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:40 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:40.821 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:40 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:40.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:40 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:40.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:40 smithi192 ceph-mon[26904]: pgmap v8: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.0 MiB/s wr, 253 op/s 2024-02-10T05:46:40.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:40 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:40.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:40 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:41.156 INFO:teuthology.orchestra.run.smithi028.stdout: 5 164029 11.72 MB/sec execute 215 sec latency 40.793 ms 2024-02-10T05:46:42.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:41 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:42.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:41 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:42.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:41 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:46:42.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:41 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:46:42.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:41 smithi028 ceph-mon[66336]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-10T05:46:42.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:41 smithi028 ceph-mon[66336]: Updating smithi192:/etc/ceph/ceph.conf 2024-02-10T05:46:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: 5 164496 11.73 MB/sec execute 216 sec latency 34.154 ms 2024-02-10T05:46:42.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:41 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:42.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:41 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:42.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:41 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:46:42.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:41 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:46:42.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:41 smithi192 ceph-mon[26904]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-10T05:46:42.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:41 smithi192 ceph-mon[26904]: Updating smithi192:/etc/ceph/ceph.conf 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: Updating smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: Updating smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: pgmap v9: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 4.2 MiB/s wr, 436 op/s 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: Updating smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: Updating smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:43.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:46:43.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:46:43.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T05:46:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: 5 164909 11.72 MB/sec execute 217 sec latency 87.435 ms 2024-02-10T05:46:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: Updating smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:46:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: Updating smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:46:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:46:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:46:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: pgmap v9: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 4.2 MiB/s wr, 436 op/s 2024-02-10T05:46:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: Updating smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:46:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: Updating smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:46:43.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:43.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:43.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:43.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:43.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:43.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:43.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:46:43.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:46:43.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:43 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-10T05:46:44.156 INFO:teuthology.orchestra.run.smithi028.stdout: 5 165469 11.70 MB/sec execute 218 sec latency 46.875 ms 2024-02-10T05:46:45.156 INFO:teuthology.orchestra.run.smithi028.stdout: 5 165932 11.71 MB/sec execute 219 sec latency 35.424 ms 2024-02-10T05:46:45.176 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:45 smithi028 ceph-mon[66336]: pgmap v10: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 3.8 MiB/s wr, 398 op/s 2024-02-10T05:46:45.176 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:45.176 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:46:45.176 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:46:45.177 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:46:45.260 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:45 smithi192 ceph-mon[26904]: pgmap v10: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 3.8 MiB/s wr, 398 op/s 2024-02-10T05:46:45.260 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:45 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:45.260 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:45 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:46:45.260 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:45 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:46:45.260 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:45 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:46:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: 5 166353 11.73 MB/sec execute 220 sec latency 30.799 ms 2024-02-10T05:46:46.198 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:46 smithi028 ceph-mon[66336]: Upgrade: Updating mon.smithi192 2024-02-10T05:46:46.198 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:46 smithi028 ceph-mon[66336]: Deploying daemon mon.smithi192 on smithi192 2024-02-10T05:46:46.199 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:46:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:46 smithi192 ceph-mon[26904]: Upgrade: Updating mon.smithi192 2024-02-10T05:46:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:46 smithi192 ceph-mon[26904]: Deploying daemon mon.smithi192 on smithi192 2024-02-10T05:46:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:46 smithi192 ceph-mon[26904]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:46:46.720 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:46 smithi192 systemd[1]: Stopping Ceph mon.smithi192 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:46:47.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:46 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192[26899]: 2024-02-10T05:46:46.718+0000 7ff1ab550700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi192 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:46:47.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:46 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192[26899]: 2024-02-10T05:46:46.718+0000 7ff1ab550700 -1 mon.smithi192@1(peon) e2 *** Got Signal Terminated *** 2024-02-10T05:46:47.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:46 smithi192 podman[52719]: 2024-02-10 05:46:46.851368669 +0000 UTC m=+0.251751804 container died cf2f3de335d9857551c02971eba2548a906005a8b6efe478a3ce553e8ec9414d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., RELEASE=HEAD, ceph=True, io.k8s.display-name=CentOS Stream 8, release=754, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, version=8) 2024-02-10T05:46:47.160 INFO:teuthology.orchestra.run.smithi028.stdout: 5 166740 11.73 MB/sec execute 221 sec latency 80.832 ms 2024-02-10T05:46:47.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:47 smithi192 podman[52719]: 2024-02-10 05:46:47.477746638 +0000 UTC m=+0.878129768 container cleanup cf2f3de335d9857551c02971eba2548a906005a8b6efe478a3ce553e8ec9414d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, RELEASE=HEAD, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.openshift.expose-services=, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, name=centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-10T05:46:47.696 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:47 smithi192 bash[52719]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192 2024-02-10T05:46:47.946 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:47 smithi192 podman[52731]: 2024-02-10 05:46:47.696134032 +0000 UTC m=+0.846547123 container remove cf2f3de335d9857551c02971eba2548a906005a8b6efe478a3ce553e8ec9414d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, release=754, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, GIT_CLEAN=True, version=8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-02-10T05:46:48.160 INFO:teuthology.orchestra.run.smithi028.stdout: 5 167061 11.72 MB/sec execute 222 sec latency 42.097 ms 2024-02-10T05:46:48.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:47 smithi192 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi192.service: Deactivated successfully. 2024-02-10T05:46:48.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:47 smithi192 systemd[1]: Stopped Ceph mon.smithi192 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:46:48.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:47 smithi192 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi192.service: Consumed 7.171s CPU time. 2024-02-10T05:46:48.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 systemd[1]: Starting Ceph mon.smithi192 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:46:48.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 podman[52863]: 2024-02-10 05:46:48.290582634 +0000 UTC m=+0.021724527 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:46:48.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 podman[52863]: 2024-02-10 05:46:48.422326349 +0000 UTC m=+0.153468226 container create 89a15b946f090f55885669e9d58a0efe49187c44f158eb4c7eb0ff5643740f50 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:46:49.160 INFO:teuthology.orchestra.run.smithi028.stdout: 5 167227 11.68 MB/sec execute 223 sec latency 58.084 ms 2024-02-10T05:46:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 podman[52863]: 2024-02-10 05:46:48.842301798 +0000 UTC m=+0.573443690 container init 89a15b946f090f55885669e9d58a0efe49187c44f158eb4c7eb0ff5643740f50 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-10T05:46:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 podman[52863]: 2024-02-10 05:46:48.846762526 +0000 UTC m=+0.577904404 container start 89a15b946f090f55885669e9d58a0efe49187c44f158eb4c7eb0ff5643740f50 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:46:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:46:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T05:46:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: pidfile_write: ignore empty --pid-file 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: load: jerasure load: lrc 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: RocksDB version: 7.9.2 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Git sha 0 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: DB SUMMARY 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: DB Session ID: I3B62ASGRSBOJQD1O0JN 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: MANIFEST file: MANIFEST-000009 size: 903 Bytes 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi192/store.db dir, Total Num: 1, files: 000024.sst 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi192/store.db: 000022.log size: 7632277 ; 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:46:49.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:46:49.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T05:46:49.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:46:49.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T05:46:49.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.env: 0x55c99202dae0 2024-02-10T05:46:49.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T05:46:49.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.info_log: 0x55c992f41760 2024-02-10T05:46:49.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.statistics: (nil) 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.use_fsync: 0 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:46:49.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.db_log_dir: 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.wal_dir: 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.write_buffer_manager: 0x55c992f15400 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:46:49.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.unordered_write: 0 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.row_cache: None 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.wal_filter: None 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:46:49.328 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.wal_compression: 0 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:46:49.329 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_open_files: -1 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:46:49.330 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Compression algorithms supported: 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: kZSTD supported: 0 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi192/store.db/MANIFEST-000009 2024-02-10T05:46:49.331 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.merge_operator: 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_filter: None 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c992f41880) 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: index_type: 0 2024-02-10T05:46:49.332 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: data_block_index_type: 0 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: index_shortening: 1 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: checksum: 4 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: no_block_cache: 0 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_cache: 0x55c992f45090 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_cache_options: 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: capacity : 536870912 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: num_shard_bits : 4 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: strict_capacity_limit : 0 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_cache_compressed: (nil) 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: persistent_cache: (nil) 2024-02-10T05:46:49.333 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_size: 4096 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_size_deviation: 10 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_restart_interval: 16 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: index_block_restart_interval: 1 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: metadata_block_size: 4096 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: partition_filters: 0 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: use_delta_encoding: 1 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: filter_policy: bloomfilter 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: whole_key_filtering: 1 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: verify_compression: 0 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: format_version: 5 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: enable_index_compression: 1 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: block_align: 0 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: max_auto_readahead_size: 262144 2024-02-10T05:46:49.334 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: prepopulate_block_cache: 0 2024-02-10T05:46:49.335 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: initial_auto_readahead_size: 8192 2024-02-10T05:46:49.335 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T05:46:49.335 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:46:49.335 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:46:49.335 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compression: NoCompression 2024-02-10T05:46:49.335 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:46:49.335 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:46:49.335 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:46:49.335 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.num_levels: 7 2024-02-10T05:46:49.335 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:46:49.335 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:46:49.336 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:46:49.337 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:46:49.338 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:46:49.339 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.ttl: 2592000 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:46:49.340 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi192/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12201, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 493ca81f-8e72-4ea0-b413-27bd089750fa 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544008880598, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544008902804, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 4605288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12206, "largest_seqno": 14297, "table_properties": {"data_size": 4596552, "index_size": 5732, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1989, "raw_key_size": 17764, "raw_average_key_size": 22, "raw_value_size": 4579512, "raw_average_value_size": 5856, "num_data_blocks": 261, "num_entries": 782, "num_filter_entries": 782, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707544008, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "493ca81f-8e72-4ea0-b413-27bd089750fa", "db_session_id": "I3B62ASGRSBOJQD1O0JN", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544008902915, "job": 1, "event": "recovery_finished"} 2024-02-10T05:46:49.341 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi192/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c993034000 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: DB pointer 0x55c993020000 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** DB Stats ** 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:46:49.342 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** Compaction Stats [default] ** 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: L0 1/0 4.39 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 440.4 0.01 0.00 1 0.010 0 0 0.0 0.0 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: L6 1/0 6.28 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Sum 2/0 10.67 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 440.4 0.01 0.00 1 0.010 0 0 0.0 0.0 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 440.4 0.01 0.00 1 0.010 0 0 0.0 0.0 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** Compaction Stats [default] ** 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 440.4 0.01 0.00 1 0.010 0 0 0.0 0.0 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:46:49.343 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Cumulative compaction: 0.00 GB write, 172.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Interval compaction: 0.00 GB write, 172.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Block cache BinnedLRUCache@0x55c992f45090#2 capacity: 512.00 MB usage: 8.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:46:49.344 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: starting mon.smithi192 rank 1 at public addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] at bind addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi192 fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: mon.smithi192@-1(???) e2 preinit fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: mon.smithi192@-1(???).mds e7 new map 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: mon.smithi192@-1(???).mds e7 print_map 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: e7 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: legacy client fscid: 1 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Filesystem 'foofs' (1) 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: fs_name foofs 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: epoch 5 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: created 2024-02-10T05:40:28.932045+0000 2024-02-10T05:46:49.345 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: modified 2024-02-10T05:40:35.175134+0000 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: tableserver 0 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: root 0 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: session_timeout 60 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: session_autoclose 300 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: max_file_size 1099511627776 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: max_xattr_size 65536 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: required_client_features {} 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: last_failure 0 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: last_failure_osd_epoch 0 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: max_mds 1 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: in 0 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: up {0=14482} 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: failed 2024-02-10T05:46:49.346 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: damaged 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: stopped 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: data_pools [3] 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: metadata_pool 2 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: inline_data disabled 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: balancer 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: bal_rank_mask -1 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: standby_count_wanted 1 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: [mds.foofs.smithi028.xhtowe{0:14482} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.28:6834/350665837,v1:172.21.15.28:6835/350665837] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: Standby daemons: 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout: [mds.foofs.smithi192.hdcbzw{-1:24275} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.192:6832/853070905,v1:172.21.15.192:6833/853070905] compat {c=[1],r=[1],i=[7ff]}] 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: mon.smithi192@-1(???).osd e74 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-10T05:46:49.347 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: mon.smithi192@-1(???).osd e74 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T05:46:49.348 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: mon.smithi192@-1(???).osd e74 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T05:46:49.348 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: mon.smithi192@-1(???).osd e74 crush map has features 432629239337189376, adjusting msgr requires 2024-02-10T05:46:49.348 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:48 smithi192 ceph-mon[52881]: mon.smithi192@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-02-10T05:46:49.348 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:49 smithi192 bash[52863]: 89a15b946f090f55885669e9d58a0efe49187c44f158eb4c7eb0ff5643740f50 2024-02-10T05:46:49.348 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:49 smithi192 systemd[1]: Started Ceph mon.smithi192 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:46:50.160 INFO:teuthology.orchestra.run.smithi028.stdout: 5 167585 11.67 MB/sec execute 224 sec latency 41.553 ms 2024-02-10T05:46:50.292 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:50 smithi028 ceph-mon[66336]: pgmap v11: 97 pgs: 97 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 5.5 MiB/s wr, 547 op/s 2024-02-10T05:46:50.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:50 smithi192 ceph-mon[52881]: pgmap v11: 97 pgs: 97 active+clean; 345 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 5.5 MiB/s wr, 547 op/s 2024-02-10T05:46:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 167916 11.65 MB/sec execute 225 sec latency 52.525 ms 2024-02-10T05:46:51.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: pgmap v13: 97 pgs: 97 active+clean; 349 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.2 MiB/s wr, 495 op/s 2024-02-10T05:46:51.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: overall HEALTH_OK 2024-02-10T05:46:51.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:46:51.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:46:51.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: mon.smithi028 calling monitor election 2024-02-10T05:46:51.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: mon.smithi192 calling monitor election 2024-02-10T05:46:51.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: mon.smithi028 is new leader, mons smithi028,smithi192 in quorum (ranks 0,1) 2024-02-10T05:46:51.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: monmap e3: 2 mons at {smithi028=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:46:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 1 up:standby 2024-02-10T05:46:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T05:46:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: mgrmap e42: smithi028.ucuqfx(active, since 20s), standbys: smithi192.hppzcm 2024-02-10T05:46:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: overall HEALTH_OK 2024-02-10T05:46:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:51 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:46:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: pgmap v13: 97 pgs: 97 active+clean; 349 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.2 MiB/s wr, 495 op/s 2024-02-10T05:46:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: overall HEALTH_OK 2024-02-10T05:46:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-02-10T05:46:51.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mon metadata", "id": "smithi192"}]: dispatch 2024-02-10T05:46:51.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: mon.smithi028 calling monitor election 2024-02-10T05:46:51.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: mon.smithi192 calling monitor election 2024-02-10T05:46:51.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: mon.smithi028 is new leader, mons smithi028,smithi192 in quorum (ranks 0,1) 2024-02-10T05:46:51.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: monmap e3: 2 mons at {smithi028=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0],smithi192=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:46:51.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 1 up:standby 2024-02-10T05:46:51.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T05:46:51.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: mgrmap e42: smithi028.ucuqfx(active, since 20s), standbys: smithi192.hppzcm 2024-02-10T05:46:51.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: overall HEALTH_OK 2024-02-10T05:46:51.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:51.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:51.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:51 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:46:52.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 168208 11.64 MB/sec execute 226 sec latency 49.561 ms 2024-02-10T05:46:53.133 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:53 smithi028 ceph-mon[66336]: pgmap v14: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.3 MiB/s wr, 688 op/s 2024-02-10T05:46:53.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 168457 11.63 MB/sec execute 227 sec latency 62.567 ms 2024-02-10T05:46:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:53 smithi192 ceph-mon[52881]: pgmap v14: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.3 MiB/s wr, 688 op/s 2024-02-10T05:46:54.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 168772 11.60 MB/sec execute 228 sec latency 67.892 ms 2024-02-10T05:46:54.971 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:54 smithi028 ceph-mon[66336]: pgmap v15: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 479 op/s 2024-02-10T05:46:55.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 169053 11.56 MB/sec execute 229 sec latency 66.893 ms 2024-02-10T05:46:55.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:54 smithi192 ceph-mon[52881]: pgmap v15: 97 pgs: 97 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 479 op/s 2024-02-10T05:46:56.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 169258 11.52 MB/sec execute 230 sec latency 51.079 ms 2024-02-10T05:46:56.275 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:56.275 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:56.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:56.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:57.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 169549 11.53 MB/sec execute 231 sec latency 59.950 ms 2024-02-10T05:46:57.294 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:57 smithi028 ceph-mon[66336]: pgmap v16: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.7 MiB/s wr, 558 op/s 2024-02-10T05:46:57.294 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:57 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:57.294 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:57 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:57.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:57 smithi192 ceph-mon[52881]: pgmap v16: 97 pgs: 97 active+clean; 337 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.7 MiB/s wr, 558 op/s 2024-02-10T05:46:57.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:57 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:57.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:57 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:46:58.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 169975 11.54 MB/sec execute 232 sec latency 62.085 ms 2024-02-10T05:46:58.838 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:46:58 smithi028 ceph-mon[66336]: pgmap v17: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.6 MiB/s wr, 506 op/s 2024-02-10T05:46:59.067 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:46:58 smithi192 ceph-mon[52881]: pgmap v17: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 4.6 MiB/s wr, 506 op/s 2024-02-10T05:46:59.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 170389 11.54 MB/sec execute 233 sec latency 41.476 ms 2024-02-10T05:47:00.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 170671 11.53 MB/sec execute 234 sec latency 44.622 ms 2024-02-10T05:47:01.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 170869 11.50 MB/sec execute 235 sec latency 54.164 ms 2024-02-10T05:47:01.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:00 smithi028 ceph-mon[66336]: pgmap v18: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.5 MiB/s wr, 368 op/s 2024-02-10T05:47:01.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:47:01.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:01.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:01.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:01.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:47:01.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:01.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:01.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:47:01.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:47:01.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:00 smithi192 ceph-mon[52881]: pgmap v18: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.5 MiB/s wr, 368 op/s 2024-02-10T05:47:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:47:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:47:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:47:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:47:01.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 171130 11.49 MB/sec execute 236 sec latency 51.349 ms 2024-02-10T05:47:02.207 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:01 smithi028 ceph-mon[66336]: Reconfiguring mon.smithi028 (monmap changed)... 2024-02-10T05:47:02.207 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:01 smithi028 ceph-mon[66336]: Reconfiguring daemon mon.smithi028 on smithi028 2024-02-10T05:47:02.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:01 smithi192 ceph-mon[52881]: Reconfiguring mon.smithi028 (monmap changed)... 2024-02-10T05:47:02.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:01 smithi192 ceph-mon[52881]: Reconfiguring daemon mon.smithi028 on smithi028 2024-02-10T05:47:03.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 171410 11.46 MB/sec execute 237 sec latency 52.354 ms 2024-02-10T05:47:03.244 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:02 smithi028 ceph-mon[66336]: pgmap v19: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 5.7 MiB/s wr, 564 op/s 2024-02-10T05:47:03.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:02 smithi192 ceph-mon[52881]: pgmap v19: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 5.7 MiB/s wr, 564 op/s 2024-02-10T05:47:04.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 171646 11.45 MB/sec execute 238 sec latency 98.992 ms 2024-02-10T05:47:04.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:04.282 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:04.282 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:04 smithi028 ceph-mon[66336]: Reconfiguring mgr.smithi028.ucuqfx (monmap changed)... 2024-02-10T05:47:04.282 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi028.ucuqfx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:47:04.282 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:47:04.282 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:04.282 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:04 smithi028 ceph-mon[66336]: Reconfiguring daemon mgr.smithi028.ucuqfx on smithi028 2024-02-10T05:47:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:04 smithi192 ceph-mon[52881]: Reconfiguring mgr.smithi028.ucuqfx (monmap changed)... 2024-02-10T05:47:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi028.ucuqfx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:47:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:47:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:04 smithi192 ceph-mon[52881]: Reconfiguring daemon mgr.smithi028.ucuqfx on smithi028 2024-02-10T05:47:05.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 171957 11.44 MB/sec execute 239 sec latency 42.677 ms 2024-02-10T05:47:05.218 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:47:05.299 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:05 smithi028 ceph-mon[66336]: pgmap v20: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 371 op/s 2024-02-10T05:47:05.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:05 smithi192 ceph-mon[52881]: pgmap v20: 97 pgs: 97 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 371 op/s 2024-02-10T05:47:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 172279 11.41 MB/sec execute 240 sec latency 47.694 ms 2024-02-10T05:47:06.305 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:06 smithi028 ceph-mon[66336]: from='client.34148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:06.306 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:06 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:06.306 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:06 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:06.306 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:06 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi028", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:47:06.306 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:06 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:06.442 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:47:06.442 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (3m) 26s ago 12m 17.7M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:47:06.442 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (12m) 26s ago 12m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:47:06.442 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (11m) 11s ago 11m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:47:06.442 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (2m) 26s ago 12m 47.7M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:47:06.442 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (6m) 26s ago 6m 37.6M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:47:06.442 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (6m) 11s ago 6m 14.2M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:47:06.442 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (2m) 26s ago 13m 475M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:47:06.442 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (74s) 11s ago 11m 449M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (54s) 26s ago 13m 31.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (18s) 11s ago 11m 24.3M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (6m) 26s ago 6m 281M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (6m) 11s ago 6m 81.4M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (2m) 26s ago 12m 13.6M - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (2m) 11s ago 11m 19.1M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (10m) 26s ago 10m 322M 1937M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (10m) 26s ago 10m 483M 1937M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (10m) 26s ago 9m 384M 1937M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (9m) 26s ago 9m 502M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (9m) 11s ago 9m 542M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (8m) 11s ago 8m 360M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (8m) 11s ago 8m 539M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (8m) 11s ago 7m 436M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:47:06.443 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (82s) 26s ago 11m 55.7M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:47:06.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:06 smithi192 ceph-mon[52881]: from='client.34148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:06.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:06 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:06.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:06 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:06.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:06 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi028", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:47:06.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:06 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:07.114 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:47:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:47:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:47:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:47:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:47:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:47:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:47:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:47:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:47:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:47:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:47:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:47:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:47:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:47:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-10T05:47:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T05:47:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:47:07.115 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:47:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 172598 11.38 MB/sec execute 241 sec latency 53.582 ms 2024-02-10T05:47:07.316 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:07 smithi028 ceph-mon[66336]: Reconfiguring crash.smithi028 (monmap changed)... 2024-02-10T05:47:07.316 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:07 smithi028 ceph-mon[66336]: Reconfiguring daemon crash.smithi028 on smithi028 2024-02-10T05:47:07.317 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:07 smithi028 ceph-mon[66336]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:07.317 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:07 smithi028 ceph-mon[66336]: pgmap v21: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 393 op/s 2024-02-10T05:47:07.317 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:07 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/3345115018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:47:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:07 smithi192 ceph-mon[52881]: Reconfiguring crash.smithi028 (monmap changed)... 2024-02-10T05:47:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:07 smithi192 ceph-mon[52881]: Reconfiguring daemon crash.smithi028 on smithi028 2024-02-10T05:47:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:07 smithi192 ceph-mon[52881]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:07 smithi192 ceph-mon[52881]: pgmap v21: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 393 op/s 2024-02-10T05:47:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:07 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/3345115018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:47:07.676 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:47:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:47:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:47:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:47:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:47:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:47:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:47:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:47:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "4/23 daemons upgraded", 2024-02-10T05:47:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading mon daemons", 2024-02-10T05:47:07.677 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:47:07.677 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:47:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: 5 172851 11.35 MB/sec execute 242 sec latency 46.472 ms 2024-02-10T05:47:08.316 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:47:08.331 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:08 smithi028 ceph-mon[66336]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:08.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:08 smithi192 ceph-mon[52881]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:09.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 173116 11.35 MB/sec execute 243 sec latency 54.763 ms 2024-02-10T05:47:09.357 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:09 smithi028 ceph-mon[66336]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:09.358 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:09 smithi028 ceph-mon[66336]: pgmap v22: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.5 MiB/s wr, 467 op/s 2024-02-10T05:47:09.358 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:09 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:09.358 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:09 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:09.358 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:09 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/3066210169' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:47:09.358 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:09 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:47:09.358 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:09 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:09.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:09 smithi192 ceph-mon[52881]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:09.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:09 smithi192 ceph-mon[52881]: pgmap v22: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.5 MiB/s wr, 467 op/s 2024-02-10T05:47:09.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:09 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:09.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:09 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:09.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:09 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/3066210169' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:47:09.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:09 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:47:09.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:09 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:10.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 173487 11.36 MB/sec execute 244 sec latency 72.689 ms 2024-02-10T05:47:10.367 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:10 smithi028 ceph-mon[66336]: Reconfiguring osd.0 (monmap changed)... 2024-02-10T05:47:10.367 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:10 smithi028 ceph-mon[66336]: Reconfiguring daemon osd.0 on smithi028 2024-02-10T05:47:10.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:10 smithi192 ceph-mon[52881]: Reconfiguring osd.0 (monmap changed)... 2024-02-10T05:47:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:10 smithi192 ceph-mon[52881]: Reconfiguring daemon osd.0 on smithi028 2024-02-10T05:47:11.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 173863 11.36 MB/sec execute 245 sec latency 48.083 ms 2024-02-10T05:47:11.388 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:11 smithi028 ceph-mon[66336]: pgmap v23: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 371 op/s 2024-02-10T05:47:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:11 smithi192 ceph-mon[52881]: pgmap v23: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 371 op/s 2024-02-10T05:47:12.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 174106 11.34 MB/sec execute 246 sec latency 36.190 ms 2024-02-10T05:47:12.658 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:12 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:12.658 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:12 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:12.658 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:12 smithi028 ceph-mon[66336]: Reconfiguring osd.1 (monmap changed)... 2024-02-10T05:47:12.658 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:12 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:47:12.658 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:12 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:12.658 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:12 smithi028 ceph-mon[66336]: Reconfiguring daemon osd.1 on smithi028 2024-02-10T05:47:12.658 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:12 smithi028 ceph-mon[66336]: pgmap v24: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.7 MiB/s wr, 554 op/s 2024-02-10T05:47:12.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:12 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:12.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:12 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:12.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:12 smithi192 ceph-mon[52881]: Reconfiguring osd.1 (monmap changed)... 2024-02-10T05:47:12.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:12 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:47:12.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:12 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:12.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:12 smithi192 ceph-mon[52881]: Reconfiguring daemon osd.1 on smithi028 2024-02-10T05:47:12.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:12 smithi192 ceph-mon[52881]: pgmap v24: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.7 MiB/s wr, 554 op/s 2024-02-10T05:47:13.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 174333 11.32 MB/sec execute 247 sec latency 75.897 ms 2024-02-10T05:47:14.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 174652 11.31 MB/sec execute 248 sec latency 70.431 ms 2024-02-10T05:47:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 174948 11.29 MB/sec execute 249 sec latency 49.242 ms 2024-02-10T05:47:15.244 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:14 smithi028 ceph-mon[66336]: pgmap v25: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.4 MiB/s wr, 357 op/s 2024-02-10T05:47:15.244 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:14 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:15.245 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:14 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:15.245 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:14 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:47:15.245 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:14 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:14 smithi192 ceph-mon[52881]: pgmap v25: 97 pgs: 97 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.4 MiB/s wr, 357 op/s 2024-02-10T05:47:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:14 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:14 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:14 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:47:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:14 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:16.007 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:15 smithi028 ceph-mon[66336]: Reconfiguring osd.2 (monmap changed)... 2024-02-10T05:47:16.007 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:15 smithi028 ceph-mon[66336]: Reconfiguring daemon osd.2 on smithi028 2024-02-10T05:47:16.007 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:47:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 175368 11.31 MB/sec execute 250 sec latency 35.401 ms 2024-02-10T05:47:16.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:15 smithi192 ceph-mon[52881]: Reconfiguring osd.2 (monmap changed)... 2024-02-10T05:47:16.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:15 smithi192 ceph-mon[52881]: Reconfiguring daemon osd.2 on smithi028 2024-02-10T05:47:16.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:47:17.015 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:16 smithi028 ceph-mon[66336]: pgmap v26: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 433 op/s 2024-02-10T05:47:17.016 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:17.016 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:17.016 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:47:17.016 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:17 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:17.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 175860 11.30 MB/sec execute 251 sec latency 35.772 ms 2024-02-10T05:47:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:16 smithi192 ceph-mon[52881]: pgmap v26: 97 pgs: 97 active+clean; 349 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 433 op/s 2024-02-10T05:47:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:47:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:17 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:18.034 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:18 smithi028 ceph-mon[66336]: Reconfiguring osd.3 (monmap changed)... 2024-02-10T05:47:18.034 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:18 smithi028 ceph-mon[66336]: Reconfiguring daemon osd.3 on smithi028 2024-02-10T05:47:18.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 176303 11.27 MB/sec execute 252 sec latency 30.809 ms 2024-02-10T05:47:18.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:18 smithi192 ceph-mon[52881]: Reconfiguring osd.3 (monmap changed)... 2024-02-10T05:47:18.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:18 smithi192 ceph-mon[52881]: Reconfiguring daemon osd.3 on smithi028 2024-02-10T05:47:19.051 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:19 smithi028 ceph-mon[66336]: pgmap v27: 97 pgs: 97 active+clean; 351 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.1 MiB/s wr, 525 op/s 2024-02-10T05:47:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 176749 11.30 MB/sec execute 253 sec latency 33.748 ms 2024-02-10T05:47:19.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:19 smithi192 ceph-mon[52881]: pgmap v27: 97 pgs: 97 active+clean; 351 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.1 MiB/s wr, 525 op/s 2024-02-10T05:47:20.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 177307 11.33 MB/sec execute 254 sec latency 30.259 ms 2024-02-10T05:47:21.080 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:20 smithi028 ceph-mon[66336]: pgmap v28: 97 pgs: 97 active+clean; 351 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.9 MiB/s wr, 372 op/s 2024-02-10T05:47:21.081 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:21.081 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:21.081 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:20 smithi028 ceph-mon[66336]: Reconfiguring mds.foofs.smithi028.xhtowe (monmap changed)... 2024-02-10T05:47:21.081 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi028.xhtowe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T05:47:21.081 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:21.081 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:20 smithi028 ceph-mon[66336]: Reconfiguring daemon mds.foofs.smithi028.xhtowe on smithi028 2024-02-10T05:47:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 177763 11.33 MB/sec execute 255 sec latency 30.407 ms 2024-02-10T05:47:21.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:20 smithi192 ceph-mon[52881]: pgmap v28: 97 pgs: 97 active+clean; 351 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.9 MiB/s wr, 372 op/s 2024-02-10T05:47:21.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:21.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:21.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:20 smithi192 ceph-mon[52881]: Reconfiguring mds.foofs.smithi028.xhtowe (monmap changed)... 2024-02-10T05:47:21.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi028.xhtowe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T05:47:21.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:21.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:20 smithi192 ceph-mon[52881]: Reconfiguring daemon mds.foofs.smithi028.xhtowe on smithi028 2024-02-10T05:47:22.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 178207 11.33 MB/sec execute 256 sec latency 27.555 ms 2024-02-10T05:47:23.095 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:22 smithi028 ceph-mon[66336]: pgmap v29: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.7 MiB/s wr, 660 op/s 2024-02-10T05:47:23.096 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:22 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:23.096 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:22 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:23.096 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:22 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:47:23.096 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:22 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:23.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 178695 11.35 MB/sec execute 257 sec latency 30.341 ms 2024-02-10T05:47:23.310 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:22 smithi192 ceph-mon[52881]: pgmap v29: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.7 MiB/s wr, 660 op/s 2024-02-10T05:47:23.310 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:22 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:23.310 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:22 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:23.310 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:22 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:47:23.310 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:22 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:24.103 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:23 smithi028 ceph-mon[66336]: Reconfiguring crash.smithi192 (monmap changed)... 2024-02-10T05:47:24.103 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:23 smithi028 ceph-mon[66336]: Reconfiguring daemon crash.smithi192 on smithi192 2024-02-10T05:47:24.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 179048 11.34 MB/sec execute 258 sec latency 40.745 ms 2024-02-10T05:47:24.274 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:23 smithi192 ceph-mon[52881]: Reconfiguring crash.smithi192 (monmap changed)... 2024-02-10T05:47:24.274 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:23 smithi192 ceph-mon[52881]: Reconfiguring daemon crash.smithi192 on smithi192 2024-02-10T05:47:25.002 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:25 smithi192 ceph-mon[52881]: pgmap v30: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.9 MiB/s wr, 476 op/s 2024-02-10T05:47:25.002 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:25.002 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:25.002 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.hppzcm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:47:25.002 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:47:25.002 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:25.078 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:25 smithi028 ceph-mon[66336]: pgmap v30: 97 pgs: 97 active+clean; 345 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.9 MiB/s wr, 476 op/s 2024-02-10T05:47:25.078 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:25.078 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:25.079 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi192.hppzcm", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:47:25.079 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:47:25.079 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 179539 11.32 MB/sec execute 259 sec latency 47.374 ms 2024-02-10T05:47:26.104 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:26 smithi028 ceph-mon[66336]: Reconfiguring mgr.smithi192.hppzcm (monmap changed)... 2024-02-10T05:47:26.104 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:26 smithi028 ceph-mon[66336]: Reconfiguring daemon mgr.smithi192.hppzcm on smithi192 2024-02-10T05:47:26.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 179930 11.30 MB/sec execute 260 sec latency 33.687 ms 2024-02-10T05:47:26.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:26 smithi192 ceph-mon[52881]: Reconfiguring mgr.smithi192.hppzcm (monmap changed)... 2024-02-10T05:47:26.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:26 smithi192 ceph-mon[52881]: Reconfiguring daemon mgr.smithi192.hppzcm on smithi192 2024-02-10T05:47:27.122 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:27 smithi028 ceph-mon[66336]: pgmap v31: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 6.2 MiB/s wr, 586 op/s 2024-02-10T05:47:27.123 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:27 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:27.123 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:27 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:27.123 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:27 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:47:27.123 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:27 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:47:27.123 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:27 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:27.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 180373 11.33 MB/sec execute 261 sec latency 47.442 ms 2024-02-10T05:47:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:27 smithi192 ceph-mon[52881]: pgmap v31: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 6.2 MiB/s wr, 586 op/s 2024-02-10T05:47:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:27 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:27 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:27 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:47:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:27 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:47:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:27 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:28.126 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:28 smithi028 ceph-mon[66336]: Reconfiguring mon.smithi192 (monmap changed)... 2024-02-10T05:47:28.126 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:28 smithi028 ceph-mon[66336]: Reconfiguring daemon mon.smithi192 on smithi192 2024-02-10T05:47:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 180886 11.34 MB/sec execute 262 sec latency 41.207 ms 2024-02-10T05:47:28.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:28 smithi192 ceph-mon[52881]: Reconfiguring mon.smithi192 (monmap changed)... 2024-02-10T05:47:28.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:28 smithi192 ceph-mon[52881]: Reconfiguring daemon mon.smithi192 on smithi192 2024-02-10T05:47:29.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:29 smithi028 ceph-mon[66336]: pgmap v32: 97 pgs: 97 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 676 op/s 2024-02-10T05:47:29.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:29.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:29.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:47:29.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:29 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 181307 11.35 MB/sec execute 263 sec latency 47.426 ms 2024-02-10T05:47:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:29 smithi192 ceph-mon[52881]: pgmap v32: 97 pgs: 97 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 676 op/s 2024-02-10T05:47:29.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:29 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:29.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:29 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:29.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:29 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:47:29.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:29 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:30.147 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:30 smithi028 ceph-mon[66336]: Reconfiguring osd.4 (monmap changed)... 2024-02-10T05:47:30.148 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:30 smithi028 ceph-mon[66336]: Reconfiguring daemon osd.4 on smithi192 2024-02-10T05:47:30.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 181687 11.35 MB/sec execute 264 sec latency 42.706 ms 2024-02-10T05:47:30.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:30 smithi192 ceph-mon[52881]: Reconfiguring osd.4 (monmap changed)... 2024-02-10T05:47:30.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:30 smithi192 ceph-mon[52881]: Reconfiguring daemon osd.4 on smithi192 2024-02-10T05:47:31.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 182166 11.36 MB/sec execute 265 sec latency 39.175 ms 2024-02-10T05:47:31.169 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:31 smithi028 ceph-mon[66336]: pgmap v33: 97 pgs: 97 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.3 MiB/s wr, 562 op/s 2024-02-10T05:47:31.169 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:47:31.169 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:31.170 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:31.170 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:47:31.170 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:31 smithi192 ceph-mon[52881]: pgmap v33: 97 pgs: 97 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.3 MiB/s wr, 562 op/s 2024-02-10T05:47:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:47:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:47:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:32.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 182602 11.36 MB/sec execute 266 sec latency 32.743 ms 2024-02-10T05:47:32.189 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:32 smithi028 ceph-mon[66336]: Reconfiguring osd.5 (monmap changed)... 2024-02-10T05:47:32.189 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:32 smithi028 ceph-mon[66336]: Reconfiguring daemon osd.5 on smithi192 2024-02-10T05:47:32.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:32 smithi192 ceph-mon[52881]: Reconfiguring osd.5 (monmap changed)... 2024-02-10T05:47:32.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:32 smithi192 ceph-mon[52881]: Reconfiguring daemon osd.5 on smithi192 2024-02-10T05:47:33.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 183029 11.34 MB/sec execute 267 sec latency 35.634 ms 2024-02-10T05:47:33.207 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:33 smithi028 ceph-mon[66336]: pgmap v34: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 819 op/s 2024-02-10T05:47:33.208 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:33 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:33.208 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:33 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:33.208 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:33 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:47:33.208 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:33 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:33.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:33 smithi192 ceph-mon[52881]: pgmap v34: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 819 op/s 2024-02-10T05:47:33.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:33 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:33.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:33 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:33.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:33 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:47:33.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:33 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:34.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 183278 11.31 MB/sec execute 268 sec latency 69.902 ms 2024-02-10T05:47:34.214 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:34 smithi028 ceph-mon[66336]: Reconfiguring osd.6 (monmap changed)... 2024-02-10T05:47:34.214 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:34 smithi028 ceph-mon[66336]: Reconfiguring daemon osd.6 on smithi192 2024-02-10T05:47:34.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:34 smithi192 ceph-mon[52881]: Reconfiguring osd.6 (monmap changed)... 2024-02-10T05:47:34.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:34 smithi192 ceph-mon[52881]: Reconfiguring daemon osd.6 on smithi192 2024-02-10T05:47:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 183571 11.30 MB/sec execute 269 sec latency 68.287 ms 2024-02-10T05:47:35.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:35 smithi028 ceph-mon[66336]: pgmap v35: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.5 MiB/s wr, 532 op/s 2024-02-10T05:47:35.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:35 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:35.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:35 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:35.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:35 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:47:35.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:35 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:35 smithi192 ceph-mon[52881]: pgmap v35: 97 pgs: 97 active+clean; 347 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.5 MiB/s wr, 532 op/s 2024-02-10T05:47:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:35 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:35 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:35 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:47:35.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:35 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:36.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 183880 11.29 MB/sec execute 270 sec latency 40.974 ms 2024-02-10T05:47:36.255 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:36 smithi028 ceph-mon[66336]: Reconfiguring osd.7 (monmap changed)... 2024-02-10T05:47:36.255 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:36 smithi028 ceph-mon[66336]: Reconfiguring daemon osd.7 on smithi192 2024-02-10T05:47:36.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:36 smithi192 ceph-mon[52881]: Reconfiguring osd.7 (monmap changed)... 2024-02-10T05:47:36.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:36 smithi192 ceph-mon[52881]: Reconfiguring daemon osd.7 on smithi192 2024-02-10T05:47:37.164 INFO:teuthology.orchestra.run.smithi028.stdout: 5 184391 11.31 MB/sec execute 271 sec latency 48.127 ms 2024-02-10T05:47:37.278 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:37 smithi028 ceph-mon[66336]: pgmap v36: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 7.0 MiB/s wr, 665 op/s 2024-02-10T05:47:37.279 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:37 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:37.279 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:37 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:37.279 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:37 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi192.hdcbzw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T05:47:37.279 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:37 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:37.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:37 smithi192 ceph-mon[52881]: pgmap v36: 97 pgs: 97 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 7.0 MiB/s wr, 665 op/s 2024-02-10T05:47:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:37 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:37 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:37 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi192.hdcbzw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T05:47:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:37 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:38.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 184854 11.32 MB/sec execute 272 sec latency 31.371 ms 2024-02-10T05:47:38.166 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:38 smithi192 ceph-mon[52881]: Reconfiguring mds.foofs.smithi192.hdcbzw (monmap changed)... 2024-02-10T05:47:38.166 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:38 smithi192 ceph-mon[52881]: Reconfiguring daemon mds.foofs.smithi192.hdcbzw on smithi192 2024-02-10T05:47:38.283 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:38 smithi028 ceph-mon[66336]: Reconfiguring mds.foofs.smithi192.hdcbzw (monmap changed)... 2024-02-10T05:47:38.284 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:38 smithi028 ceph-mon[66336]: Reconfiguring daemon mds.foofs.smithi192.hdcbzw on smithi192 2024-02-10T05:47:38.873 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:47:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 185229 11.33 MB/sec execute 273 sec latency 30.895 ms 2024-02-10T05:47:39.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:39 smithi028 ceph-mon[66336]: pgmap v37: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 6.3 MiB/s wr, 655 op/s 2024-02-10T05:47:39.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:39 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:39.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:39 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:39.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:39 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:47:39.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:39 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:47:39.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:39 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:47:39.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:39 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:39.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:39 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi028"}]: dispatch 2024-02-10T05:47:39.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:39 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi028"}]': finished 2024-02-10T05:47:39.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:39 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi192"}]: dispatch 2024-02-10T05:47:39.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:39 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi192"}]': finished 2024-02-10T05:47:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:39 smithi192 ceph-mon[52881]: pgmap v37: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 6.3 MiB/s wr, 655 op/s 2024-02-10T05:47:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:39 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:39 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:39 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:47:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:39 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:47:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:39 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:47:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:39 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:39 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi028"}]: dispatch 2024-02-10T05:47:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:39 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi028"}]': finished 2024-02-10T05:47:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:39 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi192"}]: dispatch 2024-02-10T05:47:39.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:39 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi192"}]': finished 2024-02-10T05:47:39.951 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:47:39.951 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (3m) 60s ago 13m 17.7M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:47:39.951 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (12m) 60s ago 12m 6405k - 17.2.0 e1d6a67b021e 4ba4a902827d 2024-02-10T05:47:39.951 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (12m) 44s ago 12m 6395k - 17.2.0 e1d6a67b021e 0dd1d7c0aaec 2024-02-10T05:47:39.951 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (3m) 60s ago 12m 47.7M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:47:39.951 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (7m) 60s ago 7m 37.6M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:47:39.951 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (7m) 44s ago 7m 14.2M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:47:39.951 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (2m) 60s ago 14m 475M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:47:39.951 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (108s) 44s ago 12m 449M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:47:39.952 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (87s) 60s ago 14m 31.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:47:39.952 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (51s) 44s ago 12m 24.3M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:47:39.952 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (6m) 60s ago 6m 281M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:47:39.952 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (6m) 44s ago 6m 81.4M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:47:39.952 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (3m) 60s ago 12m 13.6M - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:47:39.952 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (2m) 44s ago 12m 19.1M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:47:39.952 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (11m) 60s ago 11m 322M 1937M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:47:39.952 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (11m) 60s ago 11m 483M 1937M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:47:39.952 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (10m) 60s ago 10m 384M 1937M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:47:39.952 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (10m) 60s ago 10m 502M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:47:39.952 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (9m) 44s ago 9m 542M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:47:39.953 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (9m) 44s ago 9m 360M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:47:39.953 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (8m) 44s ago 8m 539M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:47:39.953 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (8m) 44s ago 8m 436M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:47:39.953 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (115s) 60s ago 11m 55.7M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:47:40.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 185505 11.30 MB/sec execute 274 sec latency 57.985 ms 2024-02-10T05:47:40.303 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:40 smithi028 ceph-mon[66336]: Upgrade: Setting container_image for all mon 2024-02-10T05:47:40.303 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:40 smithi028 ceph-mon[66336]: from='client.34172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:40.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:40 smithi192 ceph-mon[52881]: Upgrade: Setting container_image for all mon 2024-02-10T05:47:40.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:40 smithi192 ceph-mon[52881]: from='client.34172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:40.613 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:47:40.613 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:47:40.613 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:47:40.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:47:40.613 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:47:40.613 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:47:40.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:47:40.613 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:47:40.613 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:47:40.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:47:40.614 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:47:40.614 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:47:40.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:47:40.614 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:47:40.614 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-10T05:47:40.614 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T05:47:40.614 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:47:40.614 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:47:41.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 185775 11.29 MB/sec execute 275 sec latency 53.830 ms 2024-02-10T05:47:41.198 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:47:41.198 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:47:41.198 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:47:41.199 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:47:41.199 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:47:41.199 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:47:41.199 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:47:41.199 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:47:41.199 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "4/23 daemons upgraded", 2024-02-10T05:47:41.199 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading crash daemons", 2024-02-10T05:47:41.199 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:47:41.199 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:47:41.336 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:41 smithi028 ceph-mon[66336]: from='client.44123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:41.337 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:41 smithi028 ceph-mon[66336]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:41.337 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:41 smithi028 ceph-mon[66336]: pgmap v38: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.0 MiB/s wr, 490 op/s 2024-02-10T05:47:41.337 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:41 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/3381839871' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:47:41.337 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:41 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:41.337 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:41 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi028", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:47:41.337 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:41 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:41 smithi192 ceph-mon[52881]: from='client.44123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:41 smithi192 ceph-mon[52881]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:41 smithi192 ceph-mon[52881]: pgmap v38: 97 pgs: 97 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.0 MiB/s wr, 490 op/s 2024-02-10T05:47:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:41 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/3381839871' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:47:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:41 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:41 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi028", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:47:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:41 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:41.914 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:47:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 185983 11.28 MB/sec execute 276 sec latency 62.259 ms 2024-02-10T05:47:42.407 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:42 smithi028 ceph-mon[66336]: Upgrade: Updating crash.smithi028 (1/2) 2024-02-10T05:47:42.408 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:42 smithi028 ceph-mon[66336]: Deploying daemon crash.smithi028 on smithi028 2024-02-10T05:47:42.408 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:42 smithi028 ceph-mon[66336]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:42.408 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:42 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2293919838' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:47:42.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:42 smithi192 ceph-mon[52881]: Upgrade: Updating crash.smithi028 (1/2) 2024-02-10T05:47:42.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:42 smithi192 ceph-mon[52881]: Deploying daemon crash.smithi028 on smithi028 2024-02-10T05:47:42.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:42 smithi192 ceph-mon[52881]: from='client.34186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:47:42.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:42 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2293919838' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:47:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 186344 11.26 MB/sec execute 277 sec latency 51.864 ms 2024-02-10T05:47:43.415 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:43 smithi028 ceph-mon[66336]: pgmap v39: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 711 op/s 2024-02-10T05:47:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:43 smithi192 ceph-mon[52881]: pgmap v39: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 711 op/s 2024-02-10T05:47:44.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 186869 11.25 MB/sec execute 278 sec latency 39.829 ms 2024-02-10T05:47:45.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 187293 11.26 MB/sec execute 279 sec latency 38.626 ms 2024-02-10T05:47:45.493 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:45 smithi028 ceph-mon[66336]: pgmap v40: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-02-10T05:47:45.494 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:47:45.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:45 smithi192 ceph-mon[52881]: pgmap v40: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-02-10T05:47:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:47:46.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 187893 11.29 MB/sec execute 280 sec latency 45.613 ms 2024-02-10T05:47:47.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 188276 11.28 MB/sec execute 281 sec latency 39.951 ms 2024-02-10T05:47:47.269 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:47 smithi028 ceph-mon[66336]: pgmap v41: 97 pgs: 97 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.5 MiB/s wr, 650 op/s 2024-02-10T05:47:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:47 smithi192 ceph-mon[52881]: pgmap v41: 97 pgs: 97 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.5 MiB/s wr, 650 op/s 2024-02-10T05:47:48.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 188659 11.30 MB/sec execute 282 sec latency 31.411 ms 2024-02-10T05:47:48.788 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:48.788 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:48.789 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:48 smithi028 ceph-mon[66336]: pgmap v42: 97 pgs: 97 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 5.4 MiB/s wr, 573 op/s 2024-02-10T05:47:49.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:49.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:49.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:48 smithi192 ceph-mon[52881]: pgmap v42: 97 pgs: 97 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 5.4 MiB/s wr, 573 op/s 2024-02-10T05:47:49.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 189096 11.28 MB/sec execute 283 sec latency 30.407 ms 2024-02-10T05:47:50.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 189494 11.30 MB/sec execute 284 sec latency 51.551 ms 2024-02-10T05:47:50.576 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:50 smithi028 ceph-mon[66336]: Upgrade: Updating crash.smithi192 (2/2) 2024-02-10T05:47:50.577 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:50 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:50.577 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:50 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:47:50.577 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:50 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:50.577 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:50 smithi028 ceph-mon[66336]: Deploying daemon crash.smithi192 on smithi192 2024-02-10T05:47:50.577 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:50 smithi028 ceph-mon[66336]: pgmap v43: 97 pgs: 97 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-02-10T05:47:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:50 smithi192 ceph-mon[52881]: Upgrade: Updating crash.smithi192 (2/2) 2024-02-10T05:47:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:50 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:50 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi192", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T05:47:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:50 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:47:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:50 smithi192 ceph-mon[52881]: Deploying daemon crash.smithi192 on smithi192 2024-02-10T05:47:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:50 smithi192 ceph-mon[52881]: pgmap v43: 97 pgs: 97 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-02-10T05:47:51.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 189755 11.28 MB/sec execute 285 sec latency 85.651 ms 2024-02-10T05:47:52.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 190040 11.24 MB/sec execute 286 sec latency 71.494 ms 2024-02-10T05:47:53.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 190288 11.22 MB/sec execute 287 sec latency 77.039 ms 2024-02-10T05:47:53.168 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:53 smithi028 ceph-mon[66336]: pgmap v44: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 718 op/s 2024-02-10T05:47:53.300 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:53 smithi192 ceph-mon[52881]: pgmap v44: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 718 op/s 2024-02-10T05:47:54.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 190483 11.19 MB/sec execute 288 sec latency 77.236 ms 2024-02-10T05:47:55.045 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:55 smithi028 ceph-mon[66336]: pgmap v45: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.9 MiB/s wr, 497 op/s 2024-02-10T05:47:55.045 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:55 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:55.045 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:55 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:55.046 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:55 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:47:55.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 190712 11.18 MB/sec execute 289 sec latency 62.856 ms 2024-02-10T05:47:55.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:55 smithi192 ceph-mon[52881]: pgmap v45: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.9 MiB/s wr, 497 op/s 2024-02-10T05:47:55.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:55 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:55.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:55 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:55.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:55 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:47:56.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 191039 11.19 MB/sec execute 290 sec latency 43.005 ms 2024-02-10T05:47:57.085 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:57 smithi028 ceph-mon[66336]: pgmap v46: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.0 MiB/s wr, 621 op/s 2024-02-10T05:47:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 191436 11.19 MB/sec execute 291 sec latency 53.605 ms 2024-02-10T05:47:57.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:57 smithi192 ceph-mon[52881]: pgmap v46: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.0 MiB/s wr, 621 op/s 2024-02-10T05:47:58.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 191762 11.18 MB/sec execute 292 sec latency 47.474 ms 2024-02-10T05:47:58.885 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:58 smithi028 ceph-mon[66336]: pgmap v47: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 4.5 MiB/s wr, 465 op/s 2024-02-10T05:47:58.886 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:58 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:58.886 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:47:58 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:59.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:58 smithi192 ceph-mon[52881]: pgmap v47: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 4.5 MiB/s wr, 465 op/s 2024-02-10T05:47:59.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:58 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:59.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:47:58 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:47:59.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 191960 11.17 MB/sec execute 293 sec latency 49.160 ms 2024-02-10T05:48:00.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 192366 11.17 MB/sec execute 294 sec latency 51.123 ms 2024-02-10T05:48:00.649 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:00.649 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:00.650 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:00 smithi028 ceph-mon[66336]: pgmap v48: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.1 MiB/s wr, 409 op/s 2024-02-10T05:48:00.650 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:48:00.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:00.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:00.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:00 smithi192 ceph-mon[52881]: pgmap v48: 97 pgs: 97 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.1 MiB/s wr, 409 op/s 2024-02-10T05:48:00.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:48:01.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 192793 11.17 MB/sec execute 295 sec latency 72.252 ms 2024-02-10T05:48:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 193048 11.16 MB/sec execute 296 sec latency 41.600 ms 2024-02-10T05:48:02.685 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:02 smithi028 ceph-mon[66336]: mgrmap e43: smithi028.ucuqfx(active, since 91s), standbys: smithi192.hppzcm 2024-02-10T05:48:02.685 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:02 smithi028 ceph-mon[66336]: pgmap v49: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.4 MiB/s wr, 607 op/s 2024-02-10T05:48:02.932 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:02 smithi192 ceph-mon[52881]: mgrmap e43: smithi028.ucuqfx(active, since 91s), standbys: smithi192.hppzcm 2024-02-10T05:48:02.932 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:02 smithi192 ceph-mon[52881]: pgmap v49: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.4 MiB/s wr, 607 op/s 2024-02-10T05:48:03.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 193320 11.14 MB/sec execute 297 sec latency 59.869 ms 2024-02-10T05:48:04.165 INFO:teuthology.orchestra.run.smithi028.stdout: 5 193677 11.12 MB/sec execute 298 sec latency 59.900 ms 2024-02-10T05:48:04.535 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:04.536 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:04.536 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:04.536 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:05.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 193960 11.10 MB/sec execute 299 sec latency 61.254 ms 2024-02-10T05:48:05.307 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:05 smithi028 ceph-mon[66336]: pgmap v50: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.7 MiB/s wr, 362 op/s 2024-02-10T05:48:05.307 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:05 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:05.307 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:05 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:05.570 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:05 smithi192 ceph-mon[52881]: pgmap v50: 97 pgs: 97 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.7 MiB/s wr, 362 op/s 2024-02-10T05:48:05.570 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:05 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:05.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:05 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 194326 11.10 MB/sec execute 300 sec latency 46.829 ms 2024-02-10T05:48:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 194926 11.13 MB/sec execute 301 sec latency 37.752 ms 2024-02-10T05:48:07.343 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:07 smithi028 ceph-mon[66336]: pgmap v51: 97 pgs: 97 active+clean; 341 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 5.3 MiB/s wr, 526 op/s 2024-02-10T05:48:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:07 smithi192 ceph-mon[52881]: pgmap v51: 97 pgs: 97 active+clean; 341 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 5.3 MiB/s wr, 526 op/s 2024-02-10T05:48:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 195354 11.13 MB/sec execute 302 sec latency 44.581 ms 2024-02-10T05:48:08.872 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:08 smithi028 ceph-mon[66336]: pgmap v52: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 440 op/s 2024-02-10T05:48:09.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:08 smithi192 ceph-mon[52881]: pgmap v52: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 440 op/s 2024-02-10T05:48:09.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 195769 11.14 MB/sec execute 303 sec latency 33.731 ms 2024-02-10T05:48:10.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 196136 11.13 MB/sec execute 304 sec latency 54.799 ms 2024-02-10T05:48:11.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: pgmap v53: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 400 op/s 2024-02-10T05:48:11.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:11.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:11.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:48:11.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:48:11.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:11.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:11.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:48:11.143 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:11.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:11.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:11.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:11.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi028"}]: dispatch 2024-02-10T05:48:11.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi028"}]': finished 2024-02-10T05:48:11.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi192"}]: dispatch 2024-02-10T05:48:11.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi192"}]': finished 2024-02-10T05:48:11.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:11 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T05:48:11.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 196509 11.13 MB/sec execute 305 sec latency 45.398 ms 2024-02-10T05:48:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: pgmap v53: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 400 op/s 2024-02-10T05:48:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:48:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:48:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:48:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi028"}]: dispatch 2024-02-10T05:48:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi028"}]': finished 2024-02-10T05:48:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi192"}]: dispatch 2024-02-10T05:48:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi192"}]': finished 2024-02-10T05:48:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:11 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T05:48:12.154 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:12 smithi028 ceph-mon[66336]: Upgrade: Setting container_image for all crash 2024-02-10T05:48:12.155 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:12 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T05:48:12.155 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:12 smithi028 ceph-mon[66336]: Upgrade: osd.0 is safe to restart 2024-02-10T05:48:12.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 196841 11.13 MB/sec execute 306 sec latency 54.244 ms 2024-02-10T05:48:12.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:12 smithi192 ceph-mon[52881]: Upgrade: Setting container_image for all crash 2024-02-10T05:48:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:12 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-10T05:48:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:12 smithi192 ceph-mon[52881]: Upgrade: osd.0 is safe to restart 2024-02-10T05:48:12.556 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:48:13.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:13 smithi028 ceph-mon[66336]: pgmap v54: 97 pgs: 97 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.8 MiB/s wr, 660 op/s 2024-02-10T05:48:13.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 197249 11.11 MB/sec execute 307 sec latency 66.985 ms 2024-02-10T05:48:13.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:13 smithi192 ceph-mon[52881]: pgmap v54: 97 pgs: 97 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.8 MiB/s wr, 660 op/s 2024-02-10T05:48:13.687 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:48:13.687 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (4m) 10s ago 13m 16.1M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:48:13.687 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (26s) 10s ago 13m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:48:13.687 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (19s) 15s ago 12m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:48:13.687 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (3m) 10s ago 13m 45.8M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:48:13.687 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (7m) 10s ago 7m 37.7M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:48:13.687 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (7m) 15s ago 7m 14.4M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:48:13.687 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (3m) 10s ago 14m 496M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:48:13.687 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (2m) 15s ago 12m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (2m) 10s ago 15m 42.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (85s) 15s ago 12m 34.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (7m) 10s ago 7m 280M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (7m) 15s ago 7m 83.5M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (3m) 10s ago 13m 10.1M - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (3m) 15s ago 12m 21.1M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (12m) 10s ago 12m 398M 1937M 17.2.0 e1d6a67b021e 3b6785d0c373 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (11m) 10s ago 11m 580M 1937M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (11m) 10s ago 11m 498M 1937M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (10m) 10s ago 10m 584M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (10m) 15s ago 10m 567M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (9m) 15s ago 9m 430M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:48:13.688 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (9m) 15s ago 9m 331M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:48:13.689 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (9m) 15s ago 9m 523M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:48:13.689 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (2m) 10s ago 12m 60.6M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:48:14.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 197516 11.08 MB/sec execute 308 sec latency 60.954 ms 2024-02-10T05:48:14.185 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:14 smithi028 ceph-mon[66336]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:14.186 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:14 smithi028 ceph-mon[66336]: from='client.34198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:14.186 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:14 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:14.186 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:14 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:48:14.186 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:14 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:48:14.289 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:48:14.289 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:48:14.289 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:48:14.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:48:14.289 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:48:14.289 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:48:14.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:48:14.289 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:48:14.289 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-10T05:48:14.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:48:14.290 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:48:14.290 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:48:14.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:48:14.290 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:48:14.290 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-10T05:48:14.290 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 4 2024-02-10T05:48:14.290 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:48:14.290 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:48:14.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:14 smithi192 ceph-mon[52881]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:14.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:14 smithi192 ceph-mon[52881]: from='client.34198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:14.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:14 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:14.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:14 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:48:14.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:14 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:48:14.810 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:48:14.810 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:48:14.810 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:48:14.810 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:48:14.810 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:48:14.810 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:48:14.810 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:48:14.810 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:48:14.811 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:48:14.811 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "6/23 daemons upgraded", 2024-02-10T05:48:14.811 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T05:48:14.811 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:48:14.811 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:48:15.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 197851 11.08 MB/sec execute 309 sec latency 49.310 ms 2024-02-10T05:48:15.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:15 smithi028 ceph-mon[66336]: Upgrade: Updating osd.0 2024-02-10T05:48:15.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:15 smithi028 ceph-mon[66336]: Deploying daemon osd.0 on smithi028 2024-02-10T05:48:15.195 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:15 smithi028 ceph-mon[66336]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:15.195 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:15 smithi028 ceph-mon[66336]: pgmap v55: 97 pgs: 97 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 462 op/s 2024-02-10T05:48:15.195 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:15 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2085983312' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:15 smithi192 ceph-mon[52881]: Upgrade: Updating osd.0 2024-02-10T05:48:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:15 smithi192 ceph-mon[52881]: Deploying daemon osd.0 on smithi028 2024-02-10T05:48:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:15 smithi192 ceph-mon[52881]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:15 smithi192 ceph-mon[52881]: pgmap v55: 97 pgs: 97 active+clean; 347 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 462 op/s 2024-02-10T05:48:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:15 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2085983312' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:15.391 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:48:16.166 INFO:teuthology.orchestra.run.smithi028.stdout: 5 198238 11.09 MB/sec execute 310 sec latency 43.904 ms 2024-02-10T05:48:16.220 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:16 smithi028 ceph-mon[66336]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:16.220 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:48:16.220 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:16 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/1112225139' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:48:16.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:16 smithi192 ceph-mon[52881]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:16.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:48:16.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:16 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/1112225139' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:48:17.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 198748 11.10 MB/sec execute 311 sec latency 32.451 ms 2024-02-10T05:48:17.232 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:17 smithi028 ceph-mon[66336]: pgmap v56: 97 pgs: 97 active+clean; 338 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 6.3 MiB/s wr, 667 op/s 2024-02-10T05:48:17.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:17 smithi192 ceph-mon[52881]: pgmap v56: 97 pgs: 97 active+clean; 338 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 6.3 MiB/s wr, 667 op/s 2024-02-10T05:48:18.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 199038 11.10 MB/sec execute 312 sec latency 34.285 ms 2024-02-10T05:48:18.760 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:18 smithi028 systemd[1]: Stopping Ceph osd.0 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:48:19.023 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:18 smithi028 ceph-mon[66336]: pgmap v57: 97 pgs: 97 active+clean; 337 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 5.1 MiB/s wr, 523 op/s 2024-02-10T05:48:19.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:18 smithi192 ceph-mon[52881]: pgmap v57: 97 pgs: 97 active+clean; 337 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 5.1 MiB/s wr, 523 op/s 2024-02-10T05:48:19.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 199362 11.10 MB/sec execute 313 sec latency 46.366 ms 2024-02-10T05:48:19.537 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:19 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0[35079]: 2024-02-10T05:48:19.406+0000 7f2e842c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:48:19.537 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:19 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0[35079]: 2024-02-10T05:48:19.406+0000 7f2e842c5700 -1 osd.0 74 *** Got signal Terminated *** 2024-02-10T05:48:19.537 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:19 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0[35079]: 2024-02-10T05:48:19.406+0000 7f2e842c5700 -1 osd.0 74 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:48:20.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:19 smithi028 ceph-mon[66336]: osd.0 marked itself down and dead 2024-02-10T05:48:20.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:19 smithi192 ceph-mon[52881]: osd.0 marked itself down and dead 2024-02-10T05:48:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 199592 11.08 MB/sec execute 314 sec latency 365.052 ms 2024-02-10T05:48:20.755 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:20 smithi028 podman[72019]: 2024-02-10 05:48:20.476106067 +0000 UTC m=+1.490219714 container died 3b6785d0c373278f008f2f476b9e423721decdc15918076bcb0e874e439f79e7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, version=8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, RELEASE=HEAD, vcs-type=git, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0) 2024-02-10T05:48:20.755 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:20 smithi028 podman[72019]: 2024-02-10 05:48:20.755042131 +0000 UTC m=+1.769155776 container cleanup 3b6785d0c373278f008f2f476b9e423721decdc15918076bcb0e874e439f79e7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vcs-type=git, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, release=754, com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, maintainer=Guillaume Abrioux , version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8) 2024-02-10T05:48:21.005 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:20 smithi028 ceph-mon[66336]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:48:21.005 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:20 smithi028 ceph-mon[66336]: failed to encode map e75 with expected crc 2024-02-10T05:48:21.005 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:20 smithi028 ceph-mon[66336]: osdmap e75: 8 total, 7 up, 8 in 2024-02-10T05:48:21.006 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:20 smithi028 ceph-mon[66336]: failed to encode map e75 with expected crc 2024-02-10T05:48:21.006 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:20 smithi028 ceph-mon[66336]: pgmap v59: 97 pgs: 13 stale+active+clean, 84 active+clean; 337 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.8 MiB/s wr, 582 op/s 2024-02-10T05:48:21.006 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:20 smithi028 bash[72019]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0 2024-02-10T05:48:21.007 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:20 smithi028 podman[72037]: 2024-02-10 05:48:20.948310736 +0000 UTC m=+0.469112439 container remove 3b6785d0c373278f008f2f476b9e423721decdc15918076bcb0e874e439f79e7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, GIT_CLEAN=True, vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.openshift.expose-services=, version=8, ceph=True, architecture=x86_64, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-02-10T05:48:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:20 smithi192 ceph-mon[52881]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:48:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:20 smithi192 ceph-mon[52881]: failed to encode map e75 with expected crc 2024-02-10T05:48:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:20 smithi192 ceph-mon[52881]: osdmap e75: 8 total, 7 up, 8 in 2024-02-10T05:48:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:20 smithi192 ceph-mon[52881]: failed to encode map e75 with expected crc 2024-02-10T05:48:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:20 smithi192 ceph-mon[52881]: pgmap v59: 97 pgs: 13 stale+active+clean, 84 active+clean; 337 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.8 MiB/s wr, 582 op/s 2024-02-10T05:48:21.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 199684 11.05 MB/sec execute 315 sec latency 1072.057 ms 2024-02-10T05:48:21.521 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:21 smithi028 podman[72130]: 2024-02-10 05:48:21.48261705 +0000 UTC m=+0.112658040 container create 344effc9d9205713b17a3864197ebb8c0b823481f482361756cca6437e5d735d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T05:48:21.522 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:21 smithi028 podman[72130]: 2024-02-10 05:48:21.392052688 +0000 UTC m=+0.022093648 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:48:21.774 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:21 smithi028 podman[72130]: 2024-02-10 05:48:21.769699935 +0000 UTC m=+0.399740882 container init 344effc9d9205713b17a3864197ebb8c0b823481f482361756cca6437e5d735d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:48:21.774 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:21 smithi028 podman[72130]: 2024-02-10 05:48:21.774101244 +0000 UTC m=+0.404142191 container start 344effc9d9205713b17a3864197ebb8c0b823481f482361756cca6437e5d735d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-10T05:48:22.031 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:21 smithi028 podman[72130]: 2024-02-10 05:48:21.903246092 +0000 UTC m=+0.533287047 container attach 344effc9d9205713b17a3864197ebb8c0b823481f482361756cca6437e5d735d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-10T05:48:22.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:21 smithi028 ceph-mon[66336]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:21 smithi028 ceph-mon[66336]: osdmap e76: 8 total, 7 up, 8 in 2024-02-10T05:48:22.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:21 smithi028 ceph-mon[66336]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:21 smithi028 ceph-mon[66336]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:21 smithi028 ceph-mon[66336]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:21 smithi028 ceph-mon[66336]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:21 smithi028 ceph-mon[66336]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:21 smithi028 ceph-mon[66336]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:21 smithi028 ceph-mon[66336]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:21 smithi192 ceph-mon[52881]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:21 smithi192 ceph-mon[52881]: osdmap e76: 8 total, 7 up, 8 in 2024-02-10T05:48:22.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:21 smithi192 ceph-mon[52881]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:21 smithi192 ceph-mon[52881]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:21 smithi192 ceph-mon[52881]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:21 smithi192 ceph-mon[52881]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:21 smithi192 ceph-mon[52881]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:21 smithi192 ceph-mon[52881]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:21 smithi192 ceph-mon[52881]: failed to encode map e76 with expected crc 2024-02-10T05:48:22.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 199946 11.04 MB/sec execute 316 sec latency 87.248 ms 2024-02-10T05:48:22.303 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:22 smithi028 podman[72130]: 2024-02-10 05:48:22.158130286 +0000 UTC m=+0.788171218 container died 344effc9d9205713b17a3864197ebb8c0b823481f482361756cca6437e5d735d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux ) 2024-02-10T05:48:23.068 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:22 smithi028 ceph-mon[66336]: pgmap v61: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.0 MiB/s wr, 586 op/s; 223/1668 objects degraded (13.369%) 2024-02-10T05:48:23.068 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:22 smithi028 podman[72155]: 2024-02-10 05:48:22.951138429 +0000 UTC m=+0.782114181 container remove 344effc9d9205713b17a3864197ebb8c0b823481f482361756cca6437e5d735d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131) 2024-02-10T05:48:23.068 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:22 smithi028 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-10T05:48:23.068 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:22 smithi028 systemd[1]: Stopped Ceph osd.0 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:48:23.068 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:22 smithi028 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.0.service: Consumed 3min 43.872s CPU time. 2024-02-10T05:48:23.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 200155 11.03 MB/sec execute 317 sec latency 46.828 ms 2024-02-10T05:48:23.319 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:23 smithi028 systemd[1]: Starting Ceph osd.0 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:48:23.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:22 smithi192 ceph-mon[52881]: pgmap v61: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.0 MiB/s wr, 586 op/s; 223/1668 objects degraded (13.369%) 2024-02-10T05:48:23.569 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:23 smithi028 podman[72283]: 2024-02-10 05:48:23.419706316 +0000 UTC m=+0.024081379 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:48:23.569 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:23 smithi028 podman[72283]: 2024-02-10 05:48:23.552476615 +0000 UTC m=+0.156851658 container create 4979ed8ee992f2d7894a7c5dfb2e51885e22161f6d078cd8fe70045a3740733f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2) 2024-02-10T05:48:24.075 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:23 smithi028 ceph-mon[66336]: Health check failed: Degraded data redundancy: 223/1668 objects degraded (13.369%), 29 pgs degraded (PG_DEGRADED) 2024-02-10T05:48:24.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 200401 11.01 MB/sec execute 318 sec latency 71.791 ms 2024-02-10T05:48:24.223 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:23 smithi192 ceph-mon[52881]: Health check failed: Degraded data redundancy: 223/1668 objects degraded (13.369%), 29 pgs degraded (PG_DEGRADED) 2024-02-10T05:48:24.330 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 podman[72283]: 2024-02-10 05:48:24.080575023 +0000 UTC m=+0.684950076 container init 4979ed8ee992f2d7894a7c5dfb2e51885e22161f6d078cd8fe70045a3740733f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:48:24.331 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 podman[72283]: 2024-02-10 05:48:24.089262916 +0000 UTC m=+0.693637962 container start 4979ed8ee992f2d7894a7c5dfb2e51885e22161f6d078cd8fe70045a3740733f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-10T05:48:24.331 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 podman[72283]: 2024-02-10 05:48:24.274794996 +0000 UTC m=+0.879170065 container attach 4979ed8ee992f2d7894a7c5dfb2e51885e22161f6d078cd8fe70045a3740733f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-10T05:48:24.737 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate[72297]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T05:48:24.737 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate[72297]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T05:48:24.737 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 bash[72283]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T05:48:24.737 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 bash[72283]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T05:48:24.738 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 bash[72283]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T05:48:24.738 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 bash[72283]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T05:48:24.738 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 bash[72283]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-10T05:48:24.738 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 bash[72283]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T05:48:24.738 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 bash[72283]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-10T05:48:24.738 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate[72297]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T05:48:24.738 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate[72297]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T05:48:24.738 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate[72297]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-10T05:48:24.738 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate[72297]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-10T05:48:24.738 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate[72297]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-10T05:48:24.738 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:24 smithi028 podman[72283]: 2024-02-10 05:48:24.668319485 +0000 UTC m=+1.272694542 container died 4979ed8ee992f2d7894a7c5dfb2e51885e22161f6d078cd8fe70045a3740733f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-290212a) 2024-02-10T05:48:24.844 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:24 smithi028 ceph-mon[66336]: pgmap v62: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.4 MiB/s wr, 279 op/s; 223/1668 objects degraded (13.369%) 2024-02-10T05:48:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 200765 10.99 MB/sec execute 319 sec latency 83.232 ms 2024-02-10T05:48:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:24 smithi192 ceph-mon[52881]: pgmap v62: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 341 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.4 MiB/s wr, 279 op/s; 223/1668 objects degraded (13.369%) 2024-02-10T05:48:25.871 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:25 smithi028 podman[72419]: 2024-02-10 05:48:25.674301003 +0000 UTC m=+1.000148328 container remove 4979ed8ee992f2d7894a7c5dfb2e51885e22161f6d078cd8fe70045a3740733f (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-activate, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-10T05:48:26.128 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:25 smithi028 podman[72472]: 2024-02-10 05:48:25.831293151 +0000 UTC m=+0.022025266 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:48:26.129 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:26 smithi028 podman[72472]: 2024-02-10 05:48:26.093810747 +0000 UTC m=+0.284542854 container create 13cfdd85a4ed935dd9633f105cf572e42ff86707adec2dc8ddc6f2e629a3f0ef (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-10T05:48:26.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 201227 10.98 MB/sec execute 320 sec latency 26.481 ms 2024-02-10T05:48:26.630 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:26 smithi028 podman[72472]: 2024-02-10 05:48:26.466125259 +0000 UTC m=+0.656857372 container init 13cfdd85a4ed935dd9633f105cf572e42ff86707adec2dc8ddc6f2e629a3f0ef (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-10T05:48:26.631 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:26 smithi028 podman[72472]: 2024-02-10 05:48:26.472444012 +0000 UTC m=+0.663176119 container start 13cfdd85a4ed935dd9633f105cf572e42ff86707adec2dc8ddc6f2e629a3f0ef (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a, ceph=True, GIT_CLEAN=True) 2024-02-10T05:48:26.881 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:26 smithi028 bash[72472]: 13cfdd85a4ed935dd9633f105cf572e42ff86707adec2dc8ddc6f2e629a3f0ef 2024-02-10T05:48:26.882 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:26 smithi028 systemd[1]: Started Ceph osd.0 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:48:27.134 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:27 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0[72487]: 2024-02-10T05:48:27.070+0000 7f479996c7c0 -1 Falling back to public interface 2024-02-10T05:48:27.134 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:27 smithi028 ceph-mon[66336]: pgmap v63: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 334 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.0 KiB/s rd, 4.2 MiB/s wr, 330 op/s; 223/1671 objects degraded (13.345%) 2024-02-10T05:48:27.134 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:27 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:27.134 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:27 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:27.134 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:27 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:48:27.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 201609 10.99 MB/sec execute 321 sec latency 38.532 ms 2024-02-10T05:48:27.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:27 smithi192 ceph-mon[52881]: pgmap v63: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 334 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.0 KiB/s rd, 4.2 MiB/s wr, 330 op/s; 223/1671 objects degraded (13.345%) 2024-02-10T05:48:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:27 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:27 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:27 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:48:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 202135 11.01 MB/sec execute 322 sec latency 48.478 ms 2024-02-10T05:48:28.997 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:28 smithi028 ceph-mon[66336]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 338 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.2 MiB/s wr, 537 op/s; 217/1683 objects degraded (12.894%) 2024-02-10T05:48:29.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:28 smithi192 ceph-mon[52881]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 338 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.2 MiB/s wr, 537 op/s; 217/1683 objects degraded (12.894%) 2024-02-10T05:48:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 202605 11.02 MB/sec execute 323 sec latency 40.119 ms 2024-02-10T05:48:30.007 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:29 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 217/1683 objects degraded (12.894%), 29 pgs degraded (PG_DEGRADED) 2024-02-10T05:48:30.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:29 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 217/1683 objects degraded (12.894%), 29 pgs degraded (PG_DEGRADED) 2024-02-10T05:48:30.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 203111 11.03 MB/sec execute 324 sec latency 39.325 ms 2024-02-10T05:48:31.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 203459 11.03 MB/sec execute 325 sec latency 37.412 ms 2024-02-10T05:48:31.271 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:31 smithi028 ceph-mon[66336]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 338 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.3 MiB/s wr, 437 op/s; 217/1683 objects degraded (12.894%) 2024-02-10T05:48:31.271 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:31.271 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:48:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:31 smithi192 ceph-mon[52881]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 338 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.3 MiB/s wr, 437 op/s; 217/1683 objects degraded (12.894%) 2024-02-10T05:48:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:48:32.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 203713 11.01 MB/sec execute 326 sec latency 56.185 ms 2024-02-10T05:48:33.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:33 smithi028 ceph-mon[66336]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.1 MiB/s wr, 674 op/s; 235/1746 objects degraded (13.459%) 2024-02-10T05:48:33.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 204016 11.00 MB/sec execute 327 sec latency 52.219 ms 2024-02-10T05:48:33.312 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:33 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0[72487]: 2024-02-10T05:48:33.131+0000 7f479996c7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-02-10T05:48:33.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:33 smithi192 ceph-mon[52881]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 7.1 MiB/s wr, 674 op/s; 235/1746 objects degraded (13.459%) 2024-02-10T05:48:34.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 204352 10.98 MB/sec execute 328 sec latency 42.378 ms 2024-02-10T05:48:34.832 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:34 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0[72487]: 2024-02-10T05:48:34.639+0000 7f479996c7c0 -1 osd.0 74 log_to_monitors true 2024-02-10T05:48:35.095 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:35 smithi028 ceph-mon[66336]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.6 MiB/s wr, 460 op/s; 235/1746 objects degraded (13.459%) 2024-02-10T05:48:35.095 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:35 smithi028 ceph-mon[66336]: from='osd.0 [v2:172.21.15.28:6802/4043102785,v1:172.21.15.28:6803/4043102785]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:48:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 204689 10.96 MB/sec execute 329 sec latency 37.780 ms 2024-02-10T05:48:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:35 smithi192 ceph-mon[52881]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 348 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.6 MiB/s wr, 460 op/s; 235/1746 objects degraded (13.459%) 2024-02-10T05:48:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:35 smithi192 ceph-mon[52881]: from='osd.0 [v2:172.21.15.28:6802/4043102785,v1:172.21.15.28:6803/4043102785]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:48:35.345 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:48:35 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0[72487]: 2024-02-10T05:48:35.093+0000 7f478adc4700 -1 osd.0 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:48:36.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 205001 10.96 MB/sec execute 330 sec latency 111.154 ms 2024-02-10T05:48:36.179 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:36 smithi028 ceph-mon[66336]: from='osd.0 [v2:172.21.15.28:6802/4043102785,v1:172.21.15.28:6803/4043102785]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:48:36.179 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:36 smithi028 ceph-mon[66336]: failed to encode map e77 with expected crc 2024-02-10T05:48:36.179 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:36 smithi028 ceph-mon[66336]: osdmap e77: 8 total, 7 up, 8 in 2024-02-10T05:48:36.180 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:36 smithi028 ceph-mon[66336]: from='osd.0 [v2:172.21.15.28:6802/4043102785,v1:172.21.15.28:6803/4043102785]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:48:36.180 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:36 smithi028 ceph-mon[66336]: failed to encode map e77 with expected crc 2024-02-10T05:48:36.180 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:36 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:36.180 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:36 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:36.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:36 smithi192 ceph-mon[52881]: from='osd.0 [v2:172.21.15.28:6802/4043102785,v1:172.21.15.28:6803/4043102785]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:48:36.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:36 smithi192 ceph-mon[52881]: failed to encode map e77 with expected crc 2024-02-10T05:48:36.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:36 smithi192 ceph-mon[52881]: osdmap e77: 8 total, 7 up, 8 in 2024-02-10T05:48:36.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:36 smithi192 ceph-mon[52881]: from='osd.0 [v2:172.21.15.28:6802/4043102785,v1:172.21.15.28:6803/4043102785]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:48:36.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:36 smithi192 ceph-mon[52881]: failed to encode map e77 with expected crc 2024-02-10T05:48:36.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:36 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:36.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:36 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:37.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 205045 10.93 MB/sec execute 331 sec latency 1037.975 ms 2024-02-10T05:48:37.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:37 smithi028 ceph-mon[66336]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.2 MiB/s wr, 640 op/s; 239/1752 objects degraded (13.642%) 2024-02-10T05:48:37.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:37 smithi028 ceph-mon[66336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:48:37.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:37 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 239/1752 objects degraded (13.642%), 29 pgs degraded (PG_DEGRADED) 2024-02-10T05:48:37.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:37 smithi028 ceph-mon[66336]: failed to encode map e78 with expected crc 2024-02-10T05:48:37.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:37 smithi028 ceph-mon[66336]: osd.0 [v2:172.21.15.28:6802/4043102785,v1:172.21.15.28:6803/4043102785] boot 2024-02-10T05:48:37.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:37 smithi028 ceph-mon[66336]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T05:48:37.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:37 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:48:37.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:37 smithi028 ceph-mon[66336]: failed to encode map e78 with expected crc 2024-02-10T05:48:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:37 smithi192 ceph-mon[52881]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 346 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.2 MiB/s wr, 640 op/s; 239/1752 objects degraded (13.642%) 2024-02-10T05:48:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:37 smithi192 ceph-mon[52881]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:48:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:37 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 239/1752 objects degraded (13.642%), 29 pgs degraded (PG_DEGRADED) 2024-02-10T05:48:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:37 smithi192 ceph-mon[52881]: failed to encode map e78 with expected crc 2024-02-10T05:48:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:37 smithi192 ceph-mon[52881]: osd.0 [v2:172.21.15.28:6802/4043102785,v1:172.21.15.28:6803/4043102785] boot 2024-02-10T05:48:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:37 smithi192 ceph-mon[52881]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T05:48:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:37 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:48:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:37 smithi192 ceph-mon[52881]: failed to encode map e78 with expected crc 2024-02-10T05:48:38.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 205074 10.90 MB/sec execute 332 sec latency 2016.492 ms 2024-02-10T05:48:38.190 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:38.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:38 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:38.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T05:48:38.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: failed to encode map e79 with expected crc 2024-02-10T05:48:38.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:38.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:38 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:39.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 205657 10.93 MB/sec execute 333 sec latency 57.691 ms 2024-02-10T05:48:39.218 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:39 smithi028 ceph-mon[66336]: pgmap v72: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.7 MiB/s wr, 417 op/s; 223/1698 objects degraded (13.133%) 2024-02-10T05:48:39.219 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:39 smithi028 ceph-mon[66336]: failed to encode map e80 with expected crc 2024-02-10T05:48:39.219 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:39 smithi028 ceph-mon[66336]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T05:48:39.219 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:39 smithi028 ceph-mon[66336]: failed to encode map e80 with expected crc 2024-02-10T05:48:39.219 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:39 smithi028 ceph-mon[66336]: failed to encode map e80 with expected crc 2024-02-10T05:48:39.219 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:39 smithi028 ceph-mon[66336]: failed to encode map e80 with expected crc 2024-02-10T05:48:39.219 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:39 smithi028 ceph-mon[66336]: failed to encode map e80 with expected crc 2024-02-10T05:48:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:39 smithi192 ceph-mon[52881]: pgmap v72: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.7 MiB/s wr, 417 op/s; 223/1698 objects degraded (13.133%) 2024-02-10T05:48:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:39 smithi192 ceph-mon[52881]: failed to encode map e80 with expected crc 2024-02-10T05:48:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:39 smithi192 ceph-mon[52881]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T05:48:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:39 smithi192 ceph-mon[52881]: failed to encode map e80 with expected crc 2024-02-10T05:48:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:39 smithi192 ceph-mon[52881]: failed to encode map e80 with expected crc 2024-02-10T05:48:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:39 smithi192 ceph-mon[52881]: failed to encode map e80 with expected crc 2024-02-10T05:48:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:39 smithi192 ceph-mon[52881]: failed to encode map e80 with expected crc 2024-02-10T05:48:40.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 206064 10.93 MB/sec execute 334 sec latency 576.169 ms 2024-02-10T05:48:40.225 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:40 smithi028 ceph-mon[66336]: failed to encode map e81 with expected crc 2024-02-10T05:48:40.226 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:40 smithi028 ceph-mon[66336]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T05:48:40.226 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:40 smithi028 ceph-mon[66336]: failed to encode map e81 with expected crc 2024-02-10T05:48:40.226 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:40 smithi028 ceph-mon[66336]: failed to encode map e81 with expected crc 2024-02-10T05:48:40.226 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:40 smithi028 ceph-mon[66336]: failed to encode map e81 with expected crc 2024-02-10T05:48:40.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:40 smithi192 ceph-mon[52881]: failed to encode map e81 with expected crc 2024-02-10T05:48:40.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:40 smithi192 ceph-mon[52881]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T05:48:40.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:40 smithi192 ceph-mon[52881]: failed to encode map e81 with expected crc 2024-02-10T05:48:40.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:40 smithi192 ceph-mon[52881]: failed to encode map e81 with expected crc 2024-02-10T05:48:40.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:40 smithi192 ceph-mon[52881]: failed to encode map e81 with expected crc 2024-02-10T05:48:41.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 206469 10.94 MB/sec execute 335 sec latency 38.705 ms 2024-02-10T05:48:41.255 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:41 smithi028 ceph-mon[66336]: pgmap v75: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 240 op/s; 223/1698 objects degraded (13.133%) 2024-02-10T05:48:41.255 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:41 smithi028 ceph-mon[66336]: failed to encode map e82 with expected crc 2024-02-10T05:48:41.256 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:41 smithi028 ceph-mon[66336]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T05:48:41.256 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:41 smithi028 ceph-mon[66336]: failed to encode map e82 with expected crc 2024-02-10T05:48:41.256 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:41 smithi028 ceph-mon[66336]: failed to encode map e82 with expected crc 2024-02-10T05:48:41.256 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:41 smithi028 ceph-mon[66336]: failed to encode map e82 with expected crc 2024-02-10T05:48:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:41 smithi192 ceph-mon[52881]: pgmap v75: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 1.3 MiB/s wr, 240 op/s; 223/1698 objects degraded (13.133%) 2024-02-10T05:48:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:41 smithi192 ceph-mon[52881]: failed to encode map e82 with expected crc 2024-02-10T05:48:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:41 smithi192 ceph-mon[52881]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T05:48:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:41 smithi192 ceph-mon[52881]: failed to encode map e82 with expected crc 2024-02-10T05:48:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:41 smithi192 ceph-mon[52881]: failed to encode map e82 with expected crc 2024-02-10T05:48:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:41 smithi192 ceph-mon[52881]: failed to encode map e82 with expected crc 2024-02-10T05:48:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 206892 10.94 MB/sec execute 336 sec latency 1000.532 ms 2024-02-10T05:48:42.266 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:42 smithi028 ceph-mon[66336]: failed to encode map e83 with expected crc 2024-02-10T05:48:42.266 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:42 smithi028 ceph-mon[66336]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T05:48:42.266 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:42 smithi028 ceph-mon[66336]: failed to encode map e83 with expected crc 2024-02-10T05:48:42.267 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:42 smithi028 ceph-mon[66336]: failed to encode map e83 with expected crc 2024-02-10T05:48:42.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:42 smithi192 ceph-mon[52881]: failed to encode map e83 with expected crc 2024-02-10T05:48:42.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:42 smithi192 ceph-mon[52881]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T05:48:42.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:42 smithi192 ceph-mon[52881]: failed to encode map e83 with expected crc 2024-02-10T05:48:42.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:42 smithi192 ceph-mon[52881]: failed to encode map e83 with expected crc 2024-02-10T05:48:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 207198 10.93 MB/sec execute 337 sec latency 700.175 ms 2024-02-10T05:48:43.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: pgmap v78: 97 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 92 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.0 KiB/s rd, 5.3 MiB/s wr, 491 op/s; 65/1707 objects degraded (3.808%); 116 KiB/s, 46 objects/s recovering 2024-02-10T05:48:43.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 65/1707 objects degraded (3.808%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T05:48:43.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: failed to encode map e84 with expected crc 2024-02-10T05:48:43.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T05:48:43.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: failed to encode map e84 with expected crc 2024-02-10T05:48:43.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: failed to encode map e84 with expected crc 2024-02-10T05:48:43.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:43.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:43.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:48:43.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:48:43.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:43.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:43.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:48:43.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:43.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:43.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:43.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T05:48:43.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: failed to encode map e85 with expected crc 2024-02-10T05:48:43.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T05:48:43.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: failed to encode map e85 with expected crc 2024-02-10T05:48:43.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:43 smithi028 ceph-mon[66336]: failed to encode map e85 with expected crc 2024-02-10T05:48:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: pgmap v78: 97 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 92 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.0 KiB/s rd, 5.3 MiB/s wr, 491 op/s; 65/1707 objects degraded (3.808%); 116 KiB/s, 46 objects/s recovering 2024-02-10T05:48:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 65/1707 objects degraded (3.808%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T05:48:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: failed to encode map e84 with expected crc 2024-02-10T05:48:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T05:48:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: failed to encode map e84 with expected crc 2024-02-10T05:48:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: failed to encode map e84 with expected crc 2024-02-10T05:48:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:48:43.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:48:43.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:43.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:43.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:48:43.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:43.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:43.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:43.575 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T05:48:43.575 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: failed to encode map e85 with expected crc 2024-02-10T05:48:43.575 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T05:48:43.575 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: failed to encode map e85 with expected crc 2024-02-10T05:48:43.575 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:43 smithi192 ceph-mon[52881]: failed to encode map e85 with expected crc 2024-02-10T05:48:44.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 207473 10.92 MB/sec execute 338 sec latency 793.039 ms 2024-02-10T05:48:44.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:44 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T05:48:44.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:44 smithi028 ceph-mon[66336]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-02-10T05:48:44.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:44 smithi028 ceph-mon[66336]: failed to encode map e85 with expected crc 2024-02-10T05:48:44.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:44 smithi028 ceph-mon[66336]: failed to encode map e85 with expected crc 2024-02-10T05:48:44.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:44 smithi028 ceph-mon[66336]: failed to encode map e85 with expected crc 2024-02-10T05:48:44.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:44 smithi028 ceph-mon[66336]: failed to encode map e85 with expected crc 2024-02-10T05:48:44.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:44 smithi028 ceph-mon[66336]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T05:48:44.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:44 smithi028 ceph-mon[66336]: failed to encode map e86 with expected crc 2024-02-10T05:48:44.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:44 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T05:48:44.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:44 smithi192 ceph-mon[52881]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-02-10T05:48:44.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:44 smithi192 ceph-mon[52881]: failed to encode map e85 with expected crc 2024-02-10T05:48:44.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:44 smithi192 ceph-mon[52881]: failed to encode map e85 with expected crc 2024-02-10T05:48:44.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:44 smithi192 ceph-mon[52881]: failed to encode map e85 with expected crc 2024-02-10T05:48:44.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:44 smithi192 ceph-mon[52881]: failed to encode map e85 with expected crc 2024-02-10T05:48:44.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:44 smithi192 ceph-mon[52881]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T05:48:44.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:44 smithi192 ceph-mon[52881]: failed to encode map e86 with expected crc 2024-02-10T05:48:45.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 207705 10.89 MB/sec execute 339 sec latency 907.626 ms 2024-02-10T05:48:45.322 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:45 smithi028 ceph-mon[66336]: failed to encode map e86 with expected crc 2024-02-10T05:48:45.322 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:45 smithi028 ceph-mon[66336]: failed to encode map e86 with expected crc 2024-02-10T05:48:45.322 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:45 smithi028 ceph-mon[66336]: failed to encode map e86 with expected crc 2024-02-10T05:48:45.322 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:45 smithi028 ceph-mon[66336]: pgmap v82: 97 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 92 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.1 KiB/s rd, 5.4 MiB/s wr, 501 op/s; 65/1707 objects degraded (3.808%); 120 KiB/s, 32 objects/s recovering 2024-02-10T05:48:45.322 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:45 smithi028 ceph-mon[66336]: failed to encode map e87 with expected crc 2024-02-10T05:48:45.322 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:45 smithi028 ceph-mon[66336]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T05:48:45.322 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:45 smithi028 ceph-mon[66336]: failed to encode map e87 with expected crc 2024-02-10T05:48:45.322 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:45.322 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:48:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:45 smithi192 ceph-mon[52881]: failed to encode map e86 with expected crc 2024-02-10T05:48:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:45 smithi192 ceph-mon[52881]: failed to encode map e86 with expected crc 2024-02-10T05:48:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:45 smithi192 ceph-mon[52881]: failed to encode map e86 with expected crc 2024-02-10T05:48:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:45 smithi192 ceph-mon[52881]: pgmap v82: 97 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 92 active+clean; 343 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.1 KiB/s rd, 5.4 MiB/s wr, 501 op/s; 65/1707 objects degraded (3.808%); 120 KiB/s, 32 objects/s recovering 2024-02-10T05:48:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:45 smithi192 ceph-mon[52881]: failed to encode map e87 with expected crc 2024-02-10T05:48:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:45 smithi192 ceph-mon[52881]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T05:48:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:45 smithi192 ceph-mon[52881]: failed to encode map e87 with expected crc 2024-02-10T05:48:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:48:45.861 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:48:46.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 208004 10.87 MB/sec execute 340 sec latency 76.808 ms 2024-02-10T05:48:46.337 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:46 smithi028 ceph-mon[66336]: failed to encode map e87 with expected crc 2024-02-10T05:48:46.337 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:46 smithi028 ceph-mon[66336]: failed to encode map e87 with expected crc 2024-02-10T05:48:46.338 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:46 smithi028 ceph-mon[66336]: failed to encode map e87 with expected crc 2024-02-10T05:48:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:46 smithi192 ceph-mon[52881]: failed to encode map e87 with expected crc 2024-02-10T05:48:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:46 smithi192 ceph-mon[52881]: failed to encode map e87 with expected crc 2024-02-10T05:48:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:46 smithi192 ceph-mon[52881]: failed to encode map e87 with expected crc 2024-02-10T05:48:46.664 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:48:46.664 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (4m) 10s ago 14m 16.1M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:48:46.664 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (59s) 10s ago 14m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:48:46.664 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (52s) 48s ago 13m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:48:46.664 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (4m) 10s ago 13m 47.0M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:48:46.664 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (8m) 10s ago 8m 37.9M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (8m) 48s ago 8m 14.4M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (3m) 10s ago 15m 499M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (2m) 48s ago 13m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (2m) 10s ago 15m 44.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (118s) 48s ago 13m 34.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (7m) 10s ago 7m 278M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (7m) 48s ago 7m 83.5M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (4m) 10s ago 13m 9411k - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (3m) 48s ago 13m 21.1M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (20s) 10s ago 12m 12.0M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (12m) 10s ago 12m 351M 1937M 17.2.0 e1d6a67b021e 8dbc3d4e2d72 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (11m) 10s ago 11m 523M 1937M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:48:46.665 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (11m) 10s ago 11m 602M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:48:46.666 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (10m) 48s ago 10m 567M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:48:46.666 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (10m) 48s ago 10m 430M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:48:46.666 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (10m) 48s ago 10m 331M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:48:46.666 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (9m) 48s ago 9m 523M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:48:46.666 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (3m) 10s ago 12m 60.7M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:48:47.134 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:47 smithi028 ceph-mon[66336]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:47.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:47 smithi028 ceph-mon[66336]: pgmap v84: 97 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 4.6 MiB/s wr, 557 op/s; 23/1647 objects degraded (1.396%); 0 B/s, 10 objects/s recovering 2024-02-10T05:48:47.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:47 smithi028 ceph-mon[66336]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:47.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 208246 10.86 MB/sec execute 341 sec latency 47.350 ms 2024-02-10T05:48:47.258 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:48:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:48:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:48:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:48:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:48:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:48:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:48:47.258 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:48:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-10T05:48:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T05:48:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:48:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:48:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:48:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:48:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:48:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-02-10T05:48:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T05:48:47.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:48:47.259 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:48:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:47 smithi192 ceph-mon[52881]: from='client.34222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:47 smithi192 ceph-mon[52881]: pgmap v84: 97 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 4.6 MiB/s wr, 557 op/s; 23/1647 objects degraded (1.396%); 0 B/s, 10 objects/s recovering 2024-02-10T05:48:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:47 smithi192 ceph-mon[52881]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:47.670 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:48:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:48:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:48:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:48:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:48:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:48:47.670 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:48:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:48:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:48:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "7/23 daemons upgraded", 2024-02-10T05:48:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T05:48:47.671 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:48:47.671 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:48:48.160 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:48 smithi028 ceph-mon[66336]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:48.160 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:48 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2220638042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:48.161 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:48 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 23/1647 objects degraded (1.396%), 1 pg degraded (PG_DEGRADED) 2024-02-10T05:48:48.167 INFO:teuthology.orchestra.run.smithi028.stdout: 5 208468 10.84 MB/sec execute 342 sec latency 50.102 ms 2024-02-10T05:48:48.334 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_WARN Degraded data redundancy: 23/1647 objects degraded (1.396%), 1 pg degraded 2024-02-10T05:48:48.334 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 23/1647 objects degraded (1.396%), 1 pg degraded 2024-02-10T05:48:48.334 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.18 is active+recovery_wait+undersized+degraded+remapped, acting [3,1] 2024-02-10T05:48:48.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:48 smithi192 ceph-mon[52881]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:48.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:48 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2220638042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:48:48.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:48 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 23/1647 objects degraded (1.396%), 1 pg degraded (PG_DEGRADED) 2024-02-10T05:48:49.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 208792 10.85 MB/sec execute 343 sec latency 58.708 ms 2024-02-10T05:48:49.206 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:49 smithi028 ceph-mon[66336]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:49.207 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:49 smithi028 ceph-mon[66336]: pgmap v85: 97 pgs: 97 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.6 MiB/s wr, 406 op/s; 0 B/s, 11 objects/s recovering 2024-02-10T05:48:49.207 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:49 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/619668836' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:48:49.207 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:49 smithi028 ceph-mon[66336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/1647 objects degraded (1.396%), 1 pg degraded) 2024-02-10T05:48:49.207 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:49 smithi028 ceph-mon[66336]: Cluster is now healthy 2024-02-10T05:48:49.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:49 smithi192 ceph-mon[52881]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:48:49.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:49 smithi192 ceph-mon[52881]: pgmap v85: 97 pgs: 97 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.6 MiB/s wr, 406 op/s; 0 B/s, 11 objects/s recovering 2024-02-10T05:48:49.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:49 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/619668836' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:48:49.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:49 smithi192 ceph-mon[52881]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/1647 objects degraded (1.396%), 1 pg degraded) 2024-02-10T05:48:49.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:49 smithi192 ceph-mon[52881]: Cluster is now healthy 2024-02-10T05:48:50.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 209219 10.85 MB/sec execute 344 sec latency 47.411 ms 2024-02-10T05:48:51.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 209523 10.85 MB/sec execute 345 sec latency 49.465 ms 2024-02-10T05:48:51.251 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:51 smithi028 ceph-mon[66336]: pgmap v86: 97 pgs: 97 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 2.9 MiB/s wr, 332 op/s; 0 B/s, 9 objects/s recovering 2024-02-10T05:48:51.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:51 smithi192 ceph-mon[52881]: pgmap v86: 97 pgs: 97 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 2.9 MiB/s wr, 332 op/s; 0 B/s, 9 objects/s recovering 2024-02-10T05:48:52.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 209792 10.85 MB/sec execute 346 sec latency 58.366 ms 2024-02-10T05:48:53.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 210142 10.85 MB/sec execute 347 sec latency 30.148 ms 2024-02-10T05:48:53.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:53 smithi028 ceph-mon[66336]: pgmap v87: 97 pgs: 97 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.1 MiB/s wr, 556 op/s; 0 B/s, 7 objects/s recovering 2024-02-10T05:48:53.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:53 smithi192 ceph-mon[52881]: pgmap v87: 97 pgs: 97 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.1 MiB/s wr, 556 op/s; 0 B/s, 7 objects/s recovering 2024-02-10T05:48:54.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 210492 10.84 MB/sec execute 348 sec latency 46.614 ms 2024-02-10T05:48:55.139 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:55 smithi028 ceph-mon[66336]: pgmap v88: 97 pgs: 97 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.2 MiB/s wr, 453 op/s; 0 B/s, 6 objects/s recovering 2024-02-10T05:48:55.139 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:55 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:55.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 210737 10.83 MB/sec execute 349 sec latency 37.584 ms 2024-02-10T05:48:55.373 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:55 smithi192 ceph-mon[52881]: pgmap v88: 97 pgs: 97 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.2 MiB/s wr, 453 op/s; 0 B/s, 6 objects/s recovering 2024-02-10T05:48:55.373 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:55 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:48:56.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 211003 10.82 MB/sec execute 350 sec latency 51.182 ms 2024-02-10T05:48:57.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 211331 10.80 MB/sec execute 351 sec latency 56.030 ms 2024-02-10T05:48:57.362 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:57 smithi028 ceph-mon[66336]: pgmap v89: 97 pgs: 97 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 5.6 MiB/s wr, 517 op/s; 0 B/s, 5 objects/s recovering 2024-02-10T05:48:57.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:57 smithi192 ceph-mon[52881]: pgmap v89: 97 pgs: 97 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 5.6 MiB/s wr, 517 op/s; 0 B/s, 5 objects/s recovering 2024-02-10T05:48:58.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 211664 10.78 MB/sec execute 352 sec latency 47.714 ms 2024-02-10T05:48:58.401 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:58 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T05:48:58.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:58 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T05:48:59.162 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:59 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T05:48:59.163 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:59 smithi028 ceph-mon[66336]: Upgrade: osd.1 is safe to restart 2024-02-10T05:48:59.163 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:48:59 smithi028 ceph-mon[66336]: pgmap v90: 97 pgs: 97 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 388 op/s; 0 B/s, 1 objects/s recovering 2024-02-10T05:48:59.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 211916 10.77 MB/sec execute 353 sec latency 80.410 ms 2024-02-10T05:48:59.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:59 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-10T05:48:59.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:59 smithi192 ceph-mon[52881]: Upgrade: osd.1 is safe to restart 2024-02-10T05:48:59.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:48:59 smithi192 ceph-mon[52881]: pgmap v90: 97 pgs: 97 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 388 op/s; 0 B/s, 1 objects/s recovering 2024-02-10T05:49:00.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 212294 10.78 MB/sec execute 354 sec latency 50.147 ms 2024-02-10T05:49:00.934 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:00 smithi028 ceph-mon[66336]: Upgrade: Updating osd.1 2024-02-10T05:49:00.935 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:00.935 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:49:00.935 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:49:00.935 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:00 smithi028 ceph-mon[66336]: Deploying daemon osd.1 on smithi028 2024-02-10T05:49:00.935 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:00 smithi028 ceph-mon[66336]: pgmap v91: 97 pgs: 97 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.0 MiB/s wr, 376 op/s 2024-02-10T05:49:00.935 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:00.935 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:49:01.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 212743 10.78 MB/sec execute 355 sec latency 68.060 ms 2024-02-10T05:49:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:00 smithi192 ceph-mon[52881]: Upgrade: Updating osd.1 2024-02-10T05:49:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:49:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:49:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:00 smithi192 ceph-mon[52881]: Deploying daemon osd.1 on smithi028 2024-02-10T05:49:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:00 smithi192 ceph-mon[52881]: pgmap v91: 97 pgs: 97 active+clean; 343 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.0 MiB/s wr, 376 op/s 2024-02-10T05:49:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:49:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 213081 10.78 MB/sec execute 356 sec latency 42.926 ms 2024-02-10T05:49:03.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 213403 10.79 MB/sec execute 357 sec latency 42.286 ms 2024-02-10T05:49:03.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:03 smithi028 ceph-mon[66336]: pgmap v92: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.7 MiB/s wr, 575 op/s 2024-02-10T05:49:03.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:03 smithi192 ceph-mon[52881]: pgmap v92: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.7 MiB/s wr, 575 op/s 2024-02-10T05:49:04.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 213748 10.79 MB/sec execute 358 sec latency 58.468 ms 2024-02-10T05:49:05.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 214114 10.78 MB/sec execute 359 sec latency 48.168 ms 2024-02-10T05:49:05.260 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:05 smithi028 ceph-mon[66336]: pgmap v93: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.9 MiB/s wr, 396 op/s 2024-02-10T05:49:05.260 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:05 smithi028 systemd[1]: Stopping Ceph osd.1 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:49:05.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:05 smithi192 ceph-mon[52881]: pgmap v93: 97 pgs: 97 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.9 MiB/s wr, 396 op/s 2024-02-10T05:49:05.764 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:05 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[39144]: 2024-02-10T05:49:05.604+0000 7f6edd528700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:49:05.765 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:05 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[39144]: 2024-02-10T05:49:05.604+0000 7f6edd528700 -1 osd.1 87 *** Got signal Terminated *** 2024-02-10T05:49:05.765 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:05 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[39144]: 2024-02-10T05:49:05.604+0000 7f6edd528700 -1 osd.1 87 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:49:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 214364 10.76 MB/sec execute 360 sec latency 162.281 ms 2024-02-10T05:49:06.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:06 smithi028 ceph-mon[66336]: osd.1 marked itself down and dead 2024-02-10T05:49:06.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:06 smithi192 ceph-mon[52881]: osd.1 marked itself down and dead 2024-02-10T05:49:07.032 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:06 smithi028 podman[74604]: 2024-02-10 05:49:06.685337531 +0000 UTC m=+1.276643972 container died 8dbc3d4e2d72336564356931e6e95115cb6518ea3840e3d3a1054c3bb05a5eac (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1, RELEASE=HEAD, name=centos-stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-type=git, architecture=x86_64, version=8, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-10T05:49:07.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:07 smithi028 ceph-mon[66336]: pgmap v94: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.7 MiB/s wr, 517 op/s 2024-02-10T05:49:07.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:07 smithi028 ceph-mon[66336]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:49:07.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:07 smithi028 ceph-mon[66336]: failed to encode map e88 with expected crc 2024-02-10T05:49:07.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:07 smithi028 ceph-mon[66336]: osdmap e88: 8 total, 7 up, 8 in 2024-02-10T05:49:07.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:07 smithi028 ceph-mon[66336]: failed to encode map e88 with expected crc 2024-02-10T05:49:07.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:07 smithi028 ceph-mon[66336]: failed to encode map e88 with expected crc 2024-02-10T05:49:07.033 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:07 smithi028 ceph-mon[66336]: failed to encode map e88 with expected crc 2024-02-10T05:49:07.033 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:07 smithi028 ceph-mon[66336]: failed to encode map e88 with expected crc 2024-02-10T05:49:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:07 smithi192 ceph-mon[52881]: pgmap v94: 97 pgs: 97 active+clean; 342 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.7 MiB/s wr, 517 op/s 2024-02-10T05:49:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:07 smithi192 ceph-mon[52881]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:49:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:07 smithi192 ceph-mon[52881]: failed to encode map e88 with expected crc 2024-02-10T05:49:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:07 smithi192 ceph-mon[52881]: osdmap e88: 8 total, 7 up, 8 in 2024-02-10T05:49:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:07 smithi192 ceph-mon[52881]: failed to encode map e88 with expected crc 2024-02-10T05:49:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:07 smithi192 ceph-mon[52881]: failed to encode map e88 with expected crc 2024-02-10T05:49:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:07 smithi192 ceph-mon[52881]: failed to encode map e88 with expected crc 2024-02-10T05:49:07.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:07 smithi192 ceph-mon[52881]: failed to encode map e88 with expected crc 2024-02-10T05:49:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 214411 10.74 MB/sec execute 361 sec latency 1051.172 ms 2024-02-10T05:49:07.282 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:07 smithi028 podman[74604]: 2024-02-10 05:49:07.237624482 +0000 UTC m=+1.828930932 container cleanup 8dbc3d4e2d72336564356931e6e95115cb6518ea3840e3d3a1054c3bb05a5eac (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_BRANCH=HEAD, release=754, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, ceph=True, io.buildah.version=1.19.8, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0) 2024-02-10T05:49:07.283 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:07 smithi028 bash[74604]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1 2024-02-10T05:49:07.533 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:07 smithi028 podman[74622]: 2024-02-10 05:49:07.524216777 +0000 UTC m=+0.833980525 container remove 8dbc3d4e2d72336564356931e6e95115cb6518ea3840e3d3a1054c3bb05a5eac (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, name=centos-stream, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-10T05:49:08.035 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:08 smithi028 ceph-mon[66336]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.036 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:08 smithi028 ceph-mon[66336]: osdmap e89: 8 total, 7 up, 8 in 2024-02-10T05:49:08.036 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:08 smithi028 ceph-mon[66336]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.036 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:08 smithi028 ceph-mon[66336]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.036 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:08 smithi028 ceph-mon[66336]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.036 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:08 smithi028 ceph-mon[66336]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.036 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:08 smithi028 ceph-mon[66336]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.036 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:08 smithi028 ceph-mon[66336]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.036 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:08 smithi028 ceph-mon[66336]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.036 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:08 smithi028 ceph-mon[66336]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.036 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:08 smithi028 ceph-mon[66336]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.036 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:08 smithi028 ceph-mon[66336]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 214970 10.74 MB/sec execute 362 sec latency 38.939 ms 2024-02-10T05:49:08.287 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:08 smithi028 podman[74704]: 2024-02-10 05:49:08.05285715 +0000 UTC m=+0.033895710 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:49:08.287 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:08 smithi028 podman[74704]: 2024-02-10 05:49:08.221790301 +0000 UTC m=+0.202828845 container create 0a23ff5bef53a7e0d0e70212c60583ce34bd10b60387d693db5ba3a68f66478c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:49:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:08 smithi192 ceph-mon[52881]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:08 smithi192 ceph-mon[52881]: osdmap e89: 8 total, 7 up, 8 in 2024-02-10T05:49:08.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:08 smithi192 ceph-mon[52881]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:08 smithi192 ceph-mon[52881]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:08 smithi192 ceph-mon[52881]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:08 smithi192 ceph-mon[52881]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:08 smithi192 ceph-mon[52881]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:08 smithi192 ceph-mon[52881]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:08 smithi192 ceph-mon[52881]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:08 smithi192 ceph-mon[52881]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:08 smithi192 ceph-mon[52881]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:08 smithi192 ceph-mon[52881]: failed to encode map e89 with expected crc 2024-02-10T05:49:08.791 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:08 smithi028 podman[74704]: 2024-02-10 05:49:08.588151353 +0000 UTC m=+0.569189905 container init 0a23ff5bef53a7e0d0e70212c60583ce34bd10b60387d693db5ba3a68f66478c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=) 2024-02-10T05:49:08.791 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:08 smithi028 podman[74704]: 2024-02-10 05:49:08.596674116 +0000 UTC m=+0.577712664 container start 0a23ff5bef53a7e0d0e70212c60583ce34bd10b60387d693db5ba3a68f66478c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-02-10T05:49:08.791 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:08 smithi028 podman[74704]: 2024-02-10 05:49:08.746624941 +0000 UTC m=+0.727663487 container attach 0a23ff5bef53a7e0d0e70212c60583ce34bd10b60387d693db5ba3a68f66478c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:49:09.041 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:09 smithi028 ceph-mon[66336]: pgmap v97: 97 pgs: 13 stale+active+clean, 84 active+clean; 349 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.3 MiB/s wr, 625 op/s 2024-02-10T05:49:09.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 215453 10.73 MB/sec execute 363 sec latency 26.685 ms 2024-02-10T05:49:09.293 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:09 smithi028 podman[74704]: 2024-02-10 05:49:09.085481514 +0000 UTC m=+1.066520067 container died 0a23ff5bef53a7e0d0e70212c60583ce34bd10b60387d693db5ba3a68f66478c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:49:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:09 smithi192 ceph-mon[52881]: pgmap v97: 97 pgs: 13 stale+active+clean, 84 active+clean; 349 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.3 MiB/s wr, 625 op/s 2024-02-10T05:49:10.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 215897 10.75 MB/sec execute 364 sec latency 42.050 ms 2024-02-10T05:49:10.321 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:10 smithi028 podman[74729]: 2024-02-10 05:49:10.098180744 +0000 UTC m=+0.999252460 container remove 0a23ff5bef53a7e0d0e70212c60583ce34bd10b60387d693db5ba3a68f66478c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-10T05:49:10.321 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:10 smithi028 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-10T05:49:10.321 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:10 smithi028 systemd[1]: Stopped Ceph osd.1 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:49:10.321 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:10 smithi028 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.1.service: Consumed 3min 56.103s CPU time. 2024-02-10T05:49:10.578 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:10 smithi028 systemd[1]: Starting Ceph osd.1 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:49:10.857 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:10 smithi028 podman[74826]: 2024-02-10 05:49:10.689662926 +0000 UTC m=+0.027658367 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:49:11.109 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:11 smithi028 ceph-mon[66336]: pgmap v98: 97 pgs: 13 stale+active+clean, 84 active+clean; 349 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.8 MiB/s wr, 326 op/s 2024-02-10T05:49:11.109 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:10 smithi028 podman[74826]: 2024-02-10 05:49:10.873341686 +0000 UTC m=+0.211337173 container create b9cf029a0a5b31acf26f9d70ca2d83d41a9c325b228f5cd4e0e601574857bf6c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:49:11.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 216406 10.75 MB/sec execute 365 sec latency 47.152 ms 2024-02-10T05:49:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:11 smithi192 ceph-mon[52881]: pgmap v98: 97 pgs: 13 stale+active+clean, 84 active+clean; 349 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.8 MiB/s wr, 326 op/s 2024-02-10T05:49:11.367 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 podman[74826]: 2024-02-10 05:49:11.256843949 +0000 UTC m=+0.594839403 container init b9cf029a0a5b31acf26f9d70ca2d83d41a9c325b228f5cd4e0e601574857bf6c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-10T05:49:11.368 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 podman[74826]: 2024-02-10 05:49:11.273734349 +0000 UTC m=+0.611729790 container start b9cf029a0a5b31acf26f9d70ca2d83d41a9c325b228f5cd4e0e601574857bf6c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-10T05:49:11.619 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 podman[74826]: 2024-02-10 05:49:11.431341214 +0000 UTC m=+0.769336651 container attach b9cf029a0a5b31acf26f9d70ca2d83d41a9c325b228f5cd4e0e601574857bf6c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, ceph=True, CEPH_POINT_RELEASE=) 2024-02-10T05:49:12.127 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:12 smithi028 ceph-mon[66336]: Health check failed: Degraded data redundancy: 173/1641 objects degraded (10.542%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:12.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate[74839]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T05:49:12.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate[74839]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T05:49:12.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 bash[74826]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T05:49:12.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 bash[74826]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T05:49:12.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 bash[74826]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T05:49:12.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 bash[74826]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T05:49:12.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 bash[74826]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-10T05:49:12.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 bash[74826]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T05:49:12.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 bash[74826]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-10T05:49:12.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate[74839]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T05:49:12.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate[74839]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T05:49:12.128 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate[74839]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-10T05:49:12.129 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate[74839]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-10T05:49:12.129 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate[74839]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-10T05:49:12.129 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:11 smithi028 podman[74826]: 2024-02-10 05:49:11.969201127 +0000 UTC m=+1.307196561 container died b9cf029a0a5b31acf26f9d70ca2d83d41a9c325b228f5cd4e0e601574857bf6c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-10T05:49:12.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 216745 10.76 MB/sec execute 366 sec latency 37.307 ms 2024-02-10T05:49:12.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:12 smithi192 ceph-mon[52881]: Health check failed: Degraded data redundancy: 173/1641 objects degraded (10.542%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:13.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:13 smithi028 ceph-mon[66336]: pgmap v99: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 337 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.6 MiB/s wr, 668 op/s; 173/1641 objects degraded (10.542%) 2024-02-10T05:49:13.168 INFO:teuthology.orchestra.run.smithi028.stdout: 5 217213 10.78 MB/sec execute 367 sec latency 33.775 ms 2024-02-10T05:49:13.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:13 smithi192 ceph-mon[52881]: pgmap v99: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 337 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.6 MiB/s wr, 668 op/s; 173/1641 objects degraded (10.542%) 2024-02-10T05:49:13.651 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:13 smithi028 podman[74976]: 2024-02-10 05:49:13.625208305 +0000 UTC m=+1.644175752 container remove b9cf029a0a5b31acf26f9d70ca2d83d41a9c325b228f5cd4e0e601574857bf6c (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-activate, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:49:13.902 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:13 smithi028 podman[75026]: 2024-02-10 05:49:13.784004399 +0000 UTC m=+0.028892219 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:49:14.163 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:13 smithi028 podman[75026]: 2024-02-10 05:49:13.975397596 +0000 UTC m=+0.220285403 container create 84c53ce0260839fdca2552fb6c5eae8403c98f69f238374e7019823ad4db23f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-10T05:49:14.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 217581 10.77 MB/sec execute 368 sec latency 83.603 ms 2024-02-10T05:49:14.414 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:14 smithi028 podman[75026]: 2024-02-10 05:49:14.352094877 +0000 UTC m=+0.596982693 container init 84c53ce0260839fdca2552fb6c5eae8403c98f69f238374e7019823ad4db23f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-10T05:49:14.415 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:14 smithi028 podman[75026]: 2024-02-10 05:49:14.360928981 +0000 UTC m=+0.605816787 container start 84c53ce0260839fdca2552fb6c5eae8403c98f69f238374e7019823ad4db23f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-10T05:49:14.415 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:14 smithi028 ceph-osd[75043]: -- 172.21.15.28:0/2389095122 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b1b14d31e0 con 0x55b1b14a4c00 2024-02-10T05:49:14.682 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:14 smithi028 bash[75026]: 84c53ce0260839fdca2552fb6c5eae8403c98f69f238374e7019823ad4db23f4 2024-02-10T05:49:14.682 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:14 smithi028 systemd[1]: Started Ceph osd.1 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:49:15.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 218019 10.77 MB/sec execute 369 sec latency 37.200 ms 2024-02-10T05:49:15.183 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:14 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:49:14.948+0000 7f47d25e87c0 -1 Falling back to public interface 2024-02-10T05:49:15.183 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:15 smithi028 ceph-mon[66336]: pgmap v100: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 337 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 486 op/s; 173/1641 objects degraded (10.542%) 2024-02-10T05:49:15.184 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:15.184 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:15.184 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:49:15.184 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:15.184 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:49:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:15 smithi192 ceph-mon[52881]: pgmap v100: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 337 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 486 op/s; 173/1641 objects degraded (10.542%) 2024-02-10T05:49:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:49:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:49:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 218565 10.77 MB/sec execute 370 sec latency 35.785 ms 2024-02-10T05:49:17.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 218965 10.75 MB/sec execute 371 sec latency 52.798 ms 2024-02-10T05:49:17.212 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:17 smithi028 ceph-mon[66336]: pgmap v101: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.8 MiB/s wr, 452 op/s; 188/1740 objects degraded (10.805%) 2024-02-10T05:49:17.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:17 smithi192 ceph-mon[52881]: pgmap v101: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 339 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.8 MiB/s wr, 452 op/s; 188/1740 objects degraded (10.805%) 2024-02-10T05:49:18.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 219324 10.76 MB/sec execute 372 sec latency 65.660 ms 2024-02-10T05:49:18.228 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:18 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 188/1740 objects degraded (10.805%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:18.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:18 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 188/1740 objects degraded (10.805%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:18.891 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:49:19.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 219791 10.76 MB/sec execute 373 sec latency 46.438 ms 2024-02-10T05:49:19.262 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:19 smithi028 ceph-mon[66336]: pgmap v102: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.3 MiB/s wr, 543 op/s; 191/1767 objects degraded (10.809%) 2024-02-10T05:49:19.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:19 smithi192 ceph-mon[52881]: pgmap v102: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.3 MiB/s wr, 543 op/s; 191/1767 objects degraded (10.809%) 2024-02-10T05:49:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:49:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (5m) 44s ago 14m 16.1M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:49:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (93s) 44s ago 14m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:49:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (86s) 81s ago 14m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:49:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (4m) 44s ago 14m 47.0M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:49:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (8m) 44s ago 8m 37.9M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:49:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (8m) 81s ago 8m 14.4M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:49:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (4m) 44s ago 15m 499M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:49:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (3m) 81s ago 13m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:49:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (3m) 44s ago 16m 44.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:49:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (2m) 81s ago 13m 34.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:49:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (8m) 44s ago 8m 278M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:49:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (8m) 81s ago 8m 83.5M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:49:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (4m) 44s ago 14m 9411k - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:49:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (4m) 81s ago 13m 21.1M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:49:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (54s) 44s ago 13m 12.0M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:49:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 starting - - - 1937M 2024-02-10T05:49:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (12m) 44s ago 12m 523M 1937M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:49:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (11m) 44s ago 11m 602M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:49:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (11m) 81s ago 11m 567M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:49:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (11m) 81s ago 11m 430M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:49:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (10m) 81s ago 10m 331M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:49:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (10m) 81s ago 10m 523M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:49:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (3m) 44s ago 13m 60.7M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:49:20.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 220198 10.77 MB/sec execute 374 sec latency 37.333 ms 2024-02-10T05:49:20.282 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:20 smithi028 ceph-mon[66336]: from='client.44193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:20.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:20 smithi192 ceph-mon[52881]: from='client.44193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:20.694 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:49:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:49:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-02-10T05:49:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T05:49:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:49:20.696 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:49:21.050 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:20 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:49:20.892+0000 7f47d25e87c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-02-10T05:49:21.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 220543 10.79 MB/sec execute 375 sec latency 44.103 ms 2024-02-10T05:49:21.239 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:49:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:49:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:49:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:49:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:49:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:49:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:49:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:49:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:49:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "7/23 daemons upgraded", 2024-02-10T05:49:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T05:49:21.240 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:49:21.240 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:49:21.302 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:21 smithi028 ceph-mon[66336]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:21.302 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:21 smithi028 ceph-mon[66336]: pgmap v103: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 495 op/s; 191/1767 objects degraded (10.809%) 2024-02-10T05:49:21.303 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:21 smithi028 ceph-mon[66336]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:21.303 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:21 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/84571402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:49:21.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:21 smithi192 ceph-mon[52881]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:21.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:21 smithi192 ceph-mon[52881]: pgmap v103: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 345 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.8 MiB/s wr, 495 op/s; 191/1767 objects degraded (10.809%) 2024-02-10T05:49:21.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:21 smithi192 ceph-mon[52881]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:21.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:21 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/84571402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:49:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 191/1767 objects degraded (10.809%), 23 pgs degraded 2024-02-10T05:49:21.939 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-10T05:49:21.939 INFO:teuthology.orchestra.run.smithi028.stdout: osd.1 (root=default,host=smithi028) is down 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 191/1767 objects degraded (10.809%), 23 pgs degraded 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.4 is active+undersized+degraded, acting [0,7] 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.6 is active+undersized+degraded, acting [6,4] 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.8 is active+undersized+degraded, acting [7,0] 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.a is active+undersized+degraded, acting [3,2] 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.d is active+undersized+degraded, acting [4,3] 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.11 is active+undersized+degraded, acting [6,4] 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-02-10T05:49:21.940 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-02-10T05:49:21.941 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-02-10T05:49:21.941 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-02-10T05:49:21.941 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-02-10T05:49:21.941 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-02-10T05:49:21.941 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-02-10T05:49:21.941 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-02-10T05:49:21.941 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-02-10T05:49:21.941 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-02-10T05:49:21.941 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-02-10T05:49:21.941 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-02-10T05:49:21.941 INFO:teuthology.orchestra.run.smithi028.stdout: pg 4.a is active+undersized+degraded, acting [6,7] 2024-02-10T05:49:22.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 220832 10.78 MB/sec execute 376 sec latency 33.231 ms 2024-02-10T05:49:22.318 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:22 smithi028 ceph-mon[66336]: from='client.44209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:22.318 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:22 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2900499342' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:49:22.570 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:22 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:49:22.537+0000 7f47d25e87c0 -1 osd.1 87 log_to_monitors true 2024-02-10T05:49:22.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:22 smithi192 ceph-mon[52881]: from='client.44209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:22.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:22 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2900499342' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:49:23.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 221187 10.77 MB/sec execute 377 sec latency 48.279 ms 2024-02-10T05:49:23.344 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:23 smithi028 ceph-mon[66336]: pgmap v104: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 737 op/s; 179/1680 objects degraded (10.655%) 2024-02-10T05:49:23.345 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:23 smithi028 ceph-mon[66336]: from='osd.1 [v2:172.21.15.28:6810/314770100,v1:172.21.15.28:6811/314770100]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:49:23.345 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:23 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 179/1680 objects degraded (10.655%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:23 smithi192 ceph-mon[52881]: pgmap v104: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 340 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 737 op/s; 179/1680 objects degraded (10.655%) 2024-02-10T05:49:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:23 smithi192 ceph-mon[52881]: from='osd.1 [v2:172.21.15.28:6810/314770100,v1:172.21.15.28:6811/314770100]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:49:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:23 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 179/1680 objects degraded (10.655%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:23.595 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:49:23 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:49:23.500+0000 7f47bfa38700 -1 osd.1 87 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:49:24.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 221439 10.76 MB/sec execute 378 sec latency 76.319 ms 2024-02-10T05:49:24.351 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:24 smithi028 ceph-mon[66336]: from='osd.1 [v2:172.21.15.28:6810/314770100,v1:172.21.15.28:6811/314770100]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:49:24.351 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:24 smithi028 ceph-mon[66336]: failed to encode map e90 with expected crc 2024-02-10T05:49:24.351 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:24 smithi028 ceph-mon[66336]: osdmap e90: 8 total, 7 up, 8 in 2024-02-10T05:49:24.351 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:24 smithi028 ceph-mon[66336]: from='osd.1 [v2:172.21.15.28:6810/314770100,v1:172.21.15.28:6811/314770100]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:49:24.351 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:24 smithi028 ceph-mon[66336]: failed to encode map e90 with expected crc 2024-02-10T05:49:24.351 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:24 smithi028 ceph-mon[66336]: failed to encode map e90 with expected crc 2024-02-10T05:49:24.351 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:24 smithi028 ceph-mon[66336]: from='osd.1 [v2:172.21.15.28:6810/314770100,v1:172.21.15.28:6811/314770100]' entity='osd.1' 2024-02-10T05:49:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:24 smithi192 ceph-mon[52881]: from='osd.1 [v2:172.21.15.28:6810/314770100,v1:172.21.15.28:6811/314770100]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:49:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:24 smithi192 ceph-mon[52881]: failed to encode map e90 with expected crc 2024-02-10T05:49:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:24 smithi192 ceph-mon[52881]: osdmap e90: 8 total, 7 up, 8 in 2024-02-10T05:49:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:24 smithi192 ceph-mon[52881]: from='osd.1 [v2:172.21.15.28:6810/314770100,v1:172.21.15.28:6811/314770100]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:49:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:24 smithi192 ceph-mon[52881]: failed to encode map e90 with expected crc 2024-02-10T05:49:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:24 smithi192 ceph-mon[52881]: failed to encode map e90 with expected crc 2024-02-10T05:49:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:24 smithi192 ceph-mon[52881]: from='osd.1 [v2:172.21.15.28:6810/314770100,v1:172.21.15.28:6811/314770100]' entity='osd.1' 2024-02-10T05:49:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 221680 10.73 MB/sec execute 379 sec latency 635.045 ms 2024-02-10T05:49:25.473 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:25 smithi192 ceph-mon[52881]: pgmap v106: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.4 MiB/s wr, 611 op/s; 179/1680 objects degraded (10.655%) 2024-02-10T05:49:25.473 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:25.473 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:25.473 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:25 smithi192 ceph-mon[52881]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:49:25.473 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:25 smithi192 ceph-mon[52881]: failed to encode map e91 with expected crc 2024-02-10T05:49:25.473 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:25 smithi192 ceph-mon[52881]: osd.1 [v2:172.21.15.28:6810/314770100,v1:172.21.15.28:6811/314770100] boot 2024-02-10T05:49:25.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:25 smithi192 ceph-mon[52881]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T05:49:25.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:49:25.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:25 smithi192 ceph-mon[52881]: failed to encode map e91 with expected crc 2024-02-10T05:49:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:25 smithi028 ceph-mon[66336]: pgmap v106: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.4 MiB/s wr, 611 op/s; 179/1680 objects degraded (10.655%) 2024-02-10T05:49:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:25 smithi028 ceph-mon[66336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:49:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:25 smithi028 ceph-mon[66336]: failed to encode map e91 with expected crc 2024-02-10T05:49:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:25 smithi028 ceph-mon[66336]: osd.1 [v2:172.21.15.28:6810/314770100,v1:172.21.15.28:6811/314770100] boot 2024-02-10T05:49:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:25 smithi028 ceph-mon[66336]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T05:49:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:49:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:25 smithi028 ceph-mon[66336]: failed to encode map e91 with expected crc 2024-02-10T05:49:26.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 221680 10.70 MB/sec execute 380 sec latency 1635.089 ms 2024-02-10T05:49:26.479 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.479 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T05:49:26.479 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.479 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.479 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.479 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.479 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.479 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.479 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.480 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.480 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.480 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:26.480 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:26.480 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:26 smithi028 ceph-mon[66336]: pgmap v109: 97 pgs: 7 remapped+peering, 3 peering, 12 active+undersized+degraded, 7 active+undersized, 68 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 793 op/s; 70/1701 objects degraded (4.115%); 0 B/s, 0 objects/s recovering 2024-02-10T05:49:26.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T05:49:26.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: failed to encode map e92 with expected crc 2024-02-10T05:49:26.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:26.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:26.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:26 smithi192 ceph-mon[52881]: pgmap v109: 97 pgs: 7 remapped+peering, 3 peering, 12 active+undersized+degraded, 7 active+undersized, 68 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 793 op/s; 70/1701 objects degraded (4.115%); 0 B/s, 0 objects/s recovering 2024-02-10T05:49:27.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 221680 10.68 MB/sec execute 381 sec latency 2635.132 ms 2024-02-10T05:49:27.736 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:27 smithi028 ceph-mon[66336]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.736 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:27 smithi028 ceph-mon[66336]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T05:49:27.736 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:27 smithi028 ceph-mon[66336]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:27 smithi028 ceph-mon[66336]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:27 smithi028 ceph-mon[66336]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:27 smithi028 ceph-mon[66336]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:27 smithi028 ceph-mon[66336]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:27 smithi028 ceph-mon[66336]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:27 smithi028 ceph-mon[66336]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:27 smithi028 ceph-mon[66336]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:27 smithi192 ceph-mon[52881]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:27 smithi192 ceph-mon[52881]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T05:49:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:27 smithi192 ceph-mon[52881]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:27 smithi192 ceph-mon[52881]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:27 smithi192 ceph-mon[52881]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:27 smithi192 ceph-mon[52881]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:27 smithi192 ceph-mon[52881]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:27 smithi192 ceph-mon[52881]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:27 smithi192 ceph-mon[52881]: failed to encode map e93 with expected crc 2024-02-10T05:49:27.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:27 smithi192 ceph-mon[52881]: failed to encode map e93 with expected crc 2024-02-10T05:49:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 221869 10.67 MB/sec execute 382 sec latency 2770.206 ms 2024-02-10T05:49:28.784 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:28 smithi028 ceph-mon[66336]: failed to encode map e94 with expected crc 2024-02-10T05:49:28.785 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:28 smithi028 ceph-mon[66336]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T05:49:28.785 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:28 smithi028 ceph-mon[66336]: failed to encode map e94 with expected crc 2024-02-10T05:49:28.785 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:28 smithi028 ceph-mon[66336]: failed to encode map e94 with expected crc 2024-02-10T05:49:28.785 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:28 smithi028 ceph-mon[66336]: failed to encode map e94 with expected crc 2024-02-10T05:49:28.785 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:28 smithi028 ceph-mon[66336]: failed to encode map e94 with expected crc 2024-02-10T05:49:28.785 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:28 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 70/1701 objects degraded (4.115%), 12 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:28.785 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:28 smithi028 ceph-mon[66336]: pgmap v112: 97 pgs: 11 remapped+peering, 3 peering, 6 active+undersized+degraded, 4 active+undersized, 73 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 494 op/s; 23/1734 objects degraded (1.326%); 0 B/s, 0 objects/s recovering 2024-02-10T05:49:29.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:28 smithi192 ceph-mon[52881]: failed to encode map e94 with expected crc 2024-02-10T05:49:29.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:28 smithi192 ceph-mon[52881]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T05:49:29.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:28 smithi192 ceph-mon[52881]: failed to encode map e94 with expected crc 2024-02-10T05:49:29.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:28 smithi192 ceph-mon[52881]: failed to encode map e94 with expected crc 2024-02-10T05:49:29.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:28 smithi192 ceph-mon[52881]: failed to encode map e94 with expected crc 2024-02-10T05:49:29.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:28 smithi192 ceph-mon[52881]: failed to encode map e94 with expected crc 2024-02-10T05:49:29.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:28 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 70/1701 objects degraded (4.115%), 12 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:29.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:28 smithi192 ceph-mon[52881]: pgmap v112: 97 pgs: 11 remapped+peering, 3 peering, 6 active+undersized+degraded, 4 active+undersized, 73 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.3 MiB/s wr, 494 op/s; 23/1734 objects degraded (1.326%); 0 B/s, 0 objects/s recovering 2024-02-10T05:49:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 222121 10.65 MB/sec execute 383 sec latency 250.307 ms 2024-02-10T05:49:29.832 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:29 smithi028 ceph-mon[66336]: failed to encode map e95 with expected crc 2024-02-10T05:49:29.832 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:29 smithi028 ceph-mon[66336]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T05:49:29.832 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:29 smithi028 ceph-mon[66336]: failed to encode map e95 with expected crc 2024-02-10T05:49:29.832 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:29 smithi028 ceph-mon[66336]: failed to encode map e95 with expected crc 2024-02-10T05:49:29.832 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:29 smithi028 ceph-mon[66336]: failed to encode map e95 with expected crc 2024-02-10T05:49:29.832 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:29 smithi028 ceph-mon[66336]: failed to encode map e95 with expected crc 2024-02-10T05:49:29.832 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:29 smithi028 ceph-mon[66336]: failed to encode map e95 with expected crc 2024-02-10T05:49:29.832 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:29 smithi028 ceph-mon[66336]: failed to encode map e95 with expected crc 2024-02-10T05:49:29.832 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:29 smithi028 ceph-mon[66336]: failed to encode map e95 with expected crc 2024-02-10T05:49:29.832 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:29 smithi028 ceph-mon[66336]: failed to encode map e95 with expected crc 2024-02-10T05:49:29.832 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:29 smithi028 ceph-mon[66336]: failed to encode map e95 with expected crc 2024-02-10T05:49:30.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:29 smithi192 ceph-mon[52881]: failed to encode map e95 with expected crc 2024-02-10T05:49:30.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:29 smithi192 ceph-mon[52881]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T05:49:30.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:29 smithi192 ceph-mon[52881]: failed to encode map e95 with expected crc 2024-02-10T05:49:30.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:29 smithi192 ceph-mon[52881]: failed to encode map e95 with expected crc 2024-02-10T05:49:30.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:29 smithi192 ceph-mon[52881]: failed to encode map e95 with expected crc 2024-02-10T05:49:30.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:29 smithi192 ceph-mon[52881]: failed to encode map e95 with expected crc 2024-02-10T05:49:30.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:29 smithi192 ceph-mon[52881]: failed to encode map e95 with expected crc 2024-02-10T05:49:30.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:29 smithi192 ceph-mon[52881]: failed to encode map e95 with expected crc 2024-02-10T05:49:30.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:29 smithi192 ceph-mon[52881]: failed to encode map e95 with expected crc 2024-02-10T05:49:30.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:29 smithi192 ceph-mon[52881]: failed to encode map e95 with expected crc 2024-02-10T05:49:30.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:29 smithi192 ceph-mon[52881]: failed to encode map e95 with expected crc 2024-02-10T05:49:30.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 222398 10.64 MB/sec execute 384 sec latency 703.119 ms 2024-02-10T05:49:30.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T05:49:30.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: failed to encode map e96 with expected crc 2024-02-10T05:49:30.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: failed to encode map e96 with expected crc 2024-02-10T05:49:30.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: failed to encode map e96 with expected crc 2024-02-10T05:49:30.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: failed to encode map e96 with expected crc 2024-02-10T05:49:30.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: failed to encode map e96 with expected crc 2024-02-10T05:49:30.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: failed to encode map e96 with expected crc 2024-02-10T05:49:30.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: failed to encode map e96 with expected crc 2024-02-10T05:49:30.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: pgmap v115: 97 pgs: 11 remapped+peering, 4 peering, 82 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 451 KiB/s wr, 31 op/s; 22 B/s, 62 keys/s, 0 objects/s recovering 2024-02-10T05:49:30.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:30.853 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:49:30.854 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:30.854 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:30.854 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:49:30.854 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:49:30.854 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:30.854 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:30.854 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:49:30.854 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:49:30.854 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:49:30.854 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:49:30.854 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T05:49:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T05:49:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: failed to encode map e96 with expected crc 2024-02-10T05:49:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: failed to encode map e96 with expected crc 2024-02-10T05:49:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: failed to encode map e96 with expected crc 2024-02-10T05:49:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: failed to encode map e96 with expected crc 2024-02-10T05:49:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: failed to encode map e96 with expected crc 2024-02-10T05:49:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: failed to encode map e96 with expected crc 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: failed to encode map e96 with expected crc 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: pgmap v115: 97 pgs: 11 remapped+peering, 4 peering, 82 active+clean; 342 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 451 KiB/s wr, 31 op/s; 22 B/s, 62 keys/s, 0 objects/s recovering 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:49:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T05:49:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 222601 10.63 MB/sec execute 385 sec latency 489.003 ms 2024-02-10T05:49:31.639 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:31 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T05:49:31.640 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:31 smithi028 ceph-mon[66336]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-02-10T05:49:31.640 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:31 smithi028 ceph-mon[66336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/1734 objects degraded (1.326%), 6 pgs degraded) 2024-02-10T05:49:31.640 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:31 smithi028 ceph-mon[66336]: Cluster is now healthy 2024-02-10T05:49:31.640 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:31 smithi028 ceph-mon[66336]: failed to encode map e97 with expected crc 2024-02-10T05:49:31.640 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:31 smithi028 ceph-mon[66336]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T05:49:31.640 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:31 smithi028 ceph-mon[66336]: failed to encode map e97 with expected crc 2024-02-10T05:49:32.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:31 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T05:49:32.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:31 smithi192 ceph-mon[52881]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-02-10T05:49:32.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:31 smithi192 ceph-mon[52881]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/1734 objects degraded (1.326%), 6 pgs degraded) 2024-02-10T05:49:32.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:31 smithi192 ceph-mon[52881]: Cluster is now healthy 2024-02-10T05:49:32.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:31 smithi192 ceph-mon[52881]: failed to encode map e97 with expected crc 2024-02-10T05:49:32.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:31 smithi192 ceph-mon[52881]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T05:49:32.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:31 smithi192 ceph-mon[52881]: failed to encode map e97 with expected crc 2024-02-10T05:49:32.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 222910 10.62 MB/sec execute 386 sec latency 958.235 ms 2024-02-10T05:49:32.663 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:32 smithi028 ceph-mon[66336]: failed to encode map e98 with expected crc 2024-02-10T05:49:32.664 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:32 smithi028 ceph-mon[66336]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T05:49:32.665 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:32 smithi028 ceph-mon[66336]: failed to encode map e98 with expected crc 2024-02-10T05:49:32.665 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:32 smithi028 ceph-mon[66336]: failed to encode map e98 with expected crc 2024-02-10T05:49:32.665 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:32 smithi028 ceph-mon[66336]: failed to encode map e98 with expected crc 2024-02-10T05:49:32.665 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:32 smithi028 ceph-mon[66336]: pgmap v118: 97 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 89 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.5 MiB/s wr, 350 op/s; 104/1497 objects degraded (6.947%); 1.1 MiB/s, 62 keys/s, 40 objects/s recovering 2024-02-10T05:49:33.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:32 smithi192 ceph-mon[52881]: failed to encode map e98 with expected crc 2024-02-10T05:49:33.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:32 smithi192 ceph-mon[52881]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T05:49:33.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:32 smithi192 ceph-mon[52881]: failed to encode map e98 with expected crc 2024-02-10T05:49:33.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:32 smithi192 ceph-mon[52881]: failed to encode map e98 with expected crc 2024-02-10T05:49:33.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:32 smithi192 ceph-mon[52881]: failed to encode map e98 with expected crc 2024-02-10T05:49:33.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:32 smithi192 ceph-mon[52881]: pgmap v118: 97 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 89 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.5 MiB/s wr, 350 op/s; 104/1497 objects degraded (6.947%); 1.1 MiB/s, 62 keys/s, 40 objects/s recovering 2024-02-10T05:49:33.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 223104 10.61 MB/sec execute 387 sec latency 527.307 ms 2024-02-10T05:49:33.695 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:33 smithi028 ceph-mon[66336]: Health check failed: Degraded data redundancy: 104/1497 objects degraded (6.947%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:33.695 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:33 smithi028 ceph-mon[66336]: failed to encode map e99 with expected crc 2024-02-10T05:49:33.696 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:33 smithi028 ceph-mon[66336]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T05:49:33.696 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:33 smithi028 ceph-mon[66336]: failed to encode map e99 with expected crc 2024-02-10T05:49:33.696 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:33 smithi028 ceph-mon[66336]: failed to encode map e99 with expected crc 2024-02-10T05:49:33.696 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:33 smithi028 ceph-mon[66336]: failed to encode map e99 with expected crc 2024-02-10T05:49:33.696 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:33 smithi028 ceph-mon[66336]: failed to encode map e99 with expected crc 2024-02-10T05:49:34.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:33 smithi192 ceph-mon[52881]: Health check failed: Degraded data redundancy: 104/1497 objects degraded (6.947%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:34.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:33 smithi192 ceph-mon[52881]: failed to encode map e99 with expected crc 2024-02-10T05:49:34.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:33 smithi192 ceph-mon[52881]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T05:49:34.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:33 smithi192 ceph-mon[52881]: failed to encode map e99 with expected crc 2024-02-10T05:49:34.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:33 smithi192 ceph-mon[52881]: failed to encode map e99 with expected crc 2024-02-10T05:49:34.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:33 smithi192 ceph-mon[52881]: failed to encode map e99 with expected crc 2024-02-10T05:49:34.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:33 smithi192 ceph-mon[52881]: failed to encode map e99 with expected crc 2024-02-10T05:49:34.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 223468 10.60 MB/sec execute 388 sec latency 255.149 ms 2024-02-10T05:49:34.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:34 smithi028 ceph-mon[66336]: failed to encode map e100 with expected crc 2024-02-10T05:49:34.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:34 smithi028 ceph-mon[66336]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T05:49:34.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:34 smithi028 ceph-mon[66336]: failed to encode map e100 with expected crc 2024-02-10T05:49:34.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:34 smithi028 ceph-mon[66336]: failed to encode map e100 with expected crc 2024-02-10T05:49:34.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:34 smithi028 ceph-mon[66336]: pgmap v121: 97 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 91 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.5 MiB/s wr, 351 op/s; 104/1503 objects degraded (6.919%); 1.9 MiB/s, 40 objects/s recovering 2024-02-10T05:49:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:34 smithi192 ceph-mon[52881]: failed to encode map e100 with expected crc 2024-02-10T05:49:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:34 smithi192 ceph-mon[52881]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T05:49:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:34 smithi192 ceph-mon[52881]: failed to encode map e100 with expected crc 2024-02-10T05:49:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:34 smithi192 ceph-mon[52881]: failed to encode map e100 with expected crc 2024-02-10T05:49:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:34 smithi192 ceph-mon[52881]: pgmap v121: 97 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 91 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.5 MiB/s wr, 351 op/s; 104/1503 objects degraded (6.919%); 1.9 MiB/s, 40 objects/s recovering 2024-02-10T05:49:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 223884 10.62 MB/sec execute 389 sec latency 761.007 ms 2024-02-10T05:49:35.815 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:35 smithi028 ceph-mon[66336]: failed to encode map e101 with expected crc 2024-02-10T05:49:35.816 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:35 smithi028 ceph-mon[66336]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T05:49:35.816 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:35 smithi028 ceph-mon[66336]: failed to encode map e101 with expected crc 2024-02-10T05:49:35.816 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:35 smithi028 ceph-mon[66336]: failed to encode map e101 with expected crc 2024-02-10T05:49:35.816 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:35 smithi028 ceph-mon[66336]: failed to encode map e101 with expected crc 2024-02-10T05:49:36.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:35 smithi192 ceph-mon[52881]: failed to encode map e101 with expected crc 2024-02-10T05:49:36.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:35 smithi192 ceph-mon[52881]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T05:49:36.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:35 smithi192 ceph-mon[52881]: failed to encode map e101 with expected crc 2024-02-10T05:49:36.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:35 smithi192 ceph-mon[52881]: failed to encode map e101 with expected crc 2024-02-10T05:49:36.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:35 smithi192 ceph-mon[52881]: failed to encode map e101 with expected crc 2024-02-10T05:49:36.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 224268 10.63 MB/sec execute 390 sec latency 59.036 ms 2024-02-10T05:49:36.692 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:36 smithi028 ceph-mon[66336]: failed to encode map e101 with expected crc 2024-02-10T05:49:36.692 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:36 smithi028 ceph-mon[66336]: failed to encode map e102 with expected crc 2024-02-10T05:49:36.692 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:36 smithi028 ceph-mon[66336]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T05:49:36.692 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:36 smithi028 ceph-mon[66336]: failed to encode map e102 with expected crc 2024-02-10T05:49:36.692 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:36 smithi028 ceph-mon[66336]: failed to encode map e102 with expected crc 2024-02-10T05:49:36.692 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:36 smithi028 ceph-mon[66336]: failed to encode map e102 with expected crc 2024-02-10T05:49:36.692 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:36 smithi028 ceph-mon[66336]: failed to encode map e102 with expected crc 2024-02-10T05:49:36.692 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:36 smithi028 ceph-mon[66336]: pgmap v124: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 93 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.9 MiB/s wr, 288 op/s; 83/1614 objects degraded (5.143%); 762 KiB/s, 5 objects/s recovering 2024-02-10T05:49:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:36 smithi192 ceph-mon[52881]: failed to encode map e101 with expected crc 2024-02-10T05:49:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:36 smithi192 ceph-mon[52881]: failed to encode map e102 with expected crc 2024-02-10T05:49:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:36 smithi192 ceph-mon[52881]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T05:49:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:36 smithi192 ceph-mon[52881]: failed to encode map e102 with expected crc 2024-02-10T05:49:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:36 smithi192 ceph-mon[52881]: failed to encode map e102 with expected crc 2024-02-10T05:49:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:36 smithi192 ceph-mon[52881]: failed to encode map e102 with expected crc 2024-02-10T05:49:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:36 smithi192 ceph-mon[52881]: failed to encode map e102 with expected crc 2024-02-10T05:49:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:36 smithi192 ceph-mon[52881]: pgmap v124: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 93 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.9 MiB/s wr, 288 op/s; 83/1614 objects degraded (5.143%); 762 KiB/s, 5 objects/s recovering 2024-02-10T05:49:37.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 224590 10.63 MB/sec execute 391 sec latency 270.061 ms 2024-02-10T05:49:37.716 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:37 smithi028 ceph-mon[66336]: failed to encode map e102 with expected crc 2024-02-10T05:49:37.716 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:37 smithi028 ceph-mon[66336]: failed to encode map e103 with expected crc 2024-02-10T05:49:37.717 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:37 smithi028 ceph-mon[66336]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T05:49:37.717 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:37 smithi028 ceph-mon[66336]: failed to encode map e103 with expected crc 2024-02-10T05:49:38.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:37 smithi192 ceph-mon[52881]: failed to encode map e102 with expected crc 2024-02-10T05:49:38.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:37 smithi192 ceph-mon[52881]: failed to encode map e103 with expected crc 2024-02-10T05:49:38.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:37 smithi192 ceph-mon[52881]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T05:49:38.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:37 smithi192 ceph-mon[52881]: failed to encode map e103 with expected crc 2024-02-10T05:49:38.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 224872 10.61 MB/sec execute 392 sec latency 803.361 ms 2024-02-10T05:49:38.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:38 smithi028 ceph-mon[66336]: failed to encode map e104 with expected crc 2024-02-10T05:49:38.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:38 smithi028 ceph-mon[66336]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T05:49:38.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:38 smithi028 ceph-mon[66336]: failed to encode map e104 with expected crc 2024-02-10T05:49:38.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:38 smithi028 ceph-mon[66336]: failed to encode map e104 with expected crc 2024-02-10T05:49:38.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:38 smithi028 ceph-mon[66336]: failed to encode map e104 with expected crc 2024-02-10T05:49:38.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:38 smithi028 ceph-mon[66336]: failed to encode map e104 with expected crc 2024-02-10T05:49:38.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:38 smithi028 ceph-mon[66336]: failed to encode map e104 with expected crc 2024-02-10T05:49:38.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:38 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 83/1614 objects degraded (5.143%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:38.735 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:38 smithi028 ceph-mon[66336]: pgmap v127: 97 pgs: 1 active+undersized+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 5.0 MiB/s wr, 329 op/s; 24/1671 objects degraded (1.436%); 0 B/s, 22 objects/s recovering 2024-02-10T05:49:39.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:38 smithi192 ceph-mon[52881]: failed to encode map e104 with expected crc 2024-02-10T05:49:39.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:38 smithi192 ceph-mon[52881]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T05:49:39.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:38 smithi192 ceph-mon[52881]: failed to encode map e104 with expected crc 2024-02-10T05:49:39.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:38 smithi192 ceph-mon[52881]: failed to encode map e104 with expected crc 2024-02-10T05:49:39.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:38 smithi192 ceph-mon[52881]: failed to encode map e104 with expected crc 2024-02-10T05:49:39.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:38 smithi192 ceph-mon[52881]: failed to encode map e104 with expected crc 2024-02-10T05:49:39.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:38 smithi192 ceph-mon[52881]: failed to encode map e104 with expected crc 2024-02-10T05:49:39.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:38 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 83/1614 objects degraded (5.143%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:39.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:38 smithi192 ceph-mon[52881]: pgmap v127: 97 pgs: 1 active+undersized+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 5.0 MiB/s wr, 329 op/s; 24/1671 objects degraded (1.436%); 0 B/s, 22 objects/s recovering 2024-02-10T05:49:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 225302 10.61 MB/sec execute 393 sec latency 36.631 ms 2024-02-10T05:49:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 225729 10.60 MB/sec execute 394 sec latency 47.461 ms 2024-02-10T05:49:40.275 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:40 smithi028 ceph-mon[66336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/1671 objects degraded (1.436%), 1 pg degraded) 2024-02-10T05:49:40.275 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:40 smithi028 ceph-mon[66336]: Cluster is now healthy 2024-02-10T05:49:40.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:40 smithi192 ceph-mon[52881]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/1671 objects degraded (1.436%), 1 pg degraded) 2024-02-10T05:49:40.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:40 smithi192 ceph-mon[52881]: Cluster is now healthy 2024-02-10T05:49:41.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 226138 10.59 MB/sec execute 395 sec latency 43.828 ms 2024-02-10T05:49:41.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:41 smithi028 ceph-mon[66336]: pgmap v128: 97 pgs: 1 peering, 96 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 5.0 MiB/s wr, 465 op/s; 0 B/s, 21 objects/s recovering 2024-02-10T05:49:41.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:41 smithi192 ceph-mon[52881]: pgmap v128: 97 pgs: 1 peering, 96 active+clean; 344 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 5.0 MiB/s wr, 465 op/s; 0 B/s, 21 objects/s recovering 2024-02-10T05:49:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 226606 10.60 MB/sec execute 396 sec latency 72.828 ms 2024-02-10T05:49:43.105 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:43 smithi028 ceph-mon[66336]: pgmap v129: 97 pgs: 97 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 650 op/s; 0 B/s, 18 objects/s recovering 2024-02-10T05:49:43.169 INFO:teuthology.orchestra.run.smithi028.stdout: 5 226850 10.59 MB/sec execute 397 sec latency 59.760 ms 2024-02-10T05:49:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:43 smithi192 ceph-mon[52881]: pgmap v129: 97 pgs: 97 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 650 op/s; 0 B/s, 18 objects/s recovering 2024-02-10T05:49:44.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 227129 10.59 MB/sec execute 398 sec latency 74.222 ms 2024-02-10T05:49:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 227405 10.59 MB/sec execute 399 sec latency 62.012 ms 2024-02-10T05:49:45.172 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:45 smithi028 ceph-mon[66336]: pgmap v130: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 554 op/s; 0 B/s, 11 objects/s recovering 2024-02-10T05:49:45.172 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:45.173 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:49:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:45 smithi192 ceph-mon[52881]: pgmap v130: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 554 op/s; 0 B/s, 11 objects/s recovering 2024-02-10T05:49:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:49:46.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 227671 10.59 MB/sec execute 400 sec latency 58.236 ms 2024-02-10T05:49:46.202 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T05:49:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T05:49:47.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 227912 10.59 MB/sec execute 401 sec latency 51.962 ms 2024-02-10T05:49:47.252 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:47 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T05:49:47.252 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:47 smithi028 ceph-mon[66336]: Upgrade: osd.2 is safe to restart 2024-02-10T05:49:47.252 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:47 smithi028 ceph-mon[66336]: pgmap v131: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.3 MiB/s wr, 620 op/s; 0 B/s, 2 objects/s recovering 2024-02-10T05:49:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:47 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-10T05:49:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:47 smithi192 ceph-mon[52881]: Upgrade: osd.2 is safe to restart 2024-02-10T05:49:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:47 smithi192 ceph-mon[52881]: pgmap v131: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.3 MiB/s wr, 620 op/s; 0 B/s, 2 objects/s recovering 2024-02-10T05:49:48.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 228202 10.58 MB/sec execute 402 sec latency 58.041 ms 2024-02-10T05:49:49.046 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:48 smithi028 ceph-mon[66336]: Upgrade: Updating osd.2 2024-02-10T05:49:49.047 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:49.047 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:49:49.047 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:49:49.047 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:48 smithi028 ceph-mon[66336]: Deploying daemon osd.2 on smithi028 2024-02-10T05:49:49.047 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:48 smithi028 ceph-mon[66336]: pgmap v132: 97 pgs: 97 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 576 op/s; 0 B/s, 2 objects/s recovering 2024-02-10T05:49:49.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 228439 10.56 MB/sec execute 403 sec latency 36.225 ms 2024-02-10T05:49:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:48 smithi192 ceph-mon[52881]: Upgrade: Updating osd.2 2024-02-10T05:49:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:49:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:49:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:48 smithi192 ceph-mon[52881]: Deploying daemon osd.2 on smithi028 2024-02-10T05:49:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:48 smithi192 ceph-mon[52881]: pgmap v132: 97 pgs: 97 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 576 op/s; 0 B/s, 2 objects/s recovering 2024-02-10T05:49:50.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 228668 10.54 MB/sec execute 404 sec latency 55.310 ms 2024-02-10T05:49:51.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 228910 10.53 MB/sec execute 405 sec latency 88.949 ms 2024-02-10T05:49:51.374 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:51 smithi028 ceph-mon[66336]: pgmap v133: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.7 MiB/s wr, 501 op/s; 0 B/s, 1 objects/s recovering 2024-02-10T05:49:51.375 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:51 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:51.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:51 smithi192 ceph-mon[52881]: pgmap v133: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.7 MiB/s wr, 501 op/s; 0 B/s, 1 objects/s recovering 2024-02-10T05:49:51.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:51 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:49:52.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 229146 10.51 MB/sec execute 406 sec latency 70.152 ms 2024-02-10T05:49:52.557 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:49:52.944 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:52 smithi028 systemd[1]: Stopping Ceph osd.2 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:49:53.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 229398 10.50 MB/sec execute 407 sec latency 70.526 ms 2024-02-10T05:49:53.205 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:53 smithi028 ceph-mon[66336]: pgmap v134: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 5.7 MiB/s wr, 566 op/s 2024-02-10T05:49:53.206 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:53 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2[42852]: 2024-02-10T05:49:53.147+0000 7f007404a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:49:53.206 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:53 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2[42852]: 2024-02-10T05:49:53.147+0000 7f007404a700 -1 osd.2 104 *** Got signal Terminated *** 2024-02-10T05:49:53.206 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:53 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2[42852]: 2024-02-10T05:49:53.147+0000 7f007404a700 -1 osd.2 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:49:53.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:53 smithi192 ceph-mon[52881]: pgmap v134: 97 pgs: 97 active+clean; 340 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 5.7 MiB/s wr, 566 op/s 2024-02-10T05:49:53.715 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:49:53.715 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (6m) 29s ago 15m 15.3M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:49:53.715 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (2m) 29s ago 15m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:49:53.715 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (119s) 115s ago 14m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:49:53.715 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (5m) 29s ago 14m 46.9M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:49:53.715 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (9m) 29s ago 9m 38.0M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:49:53.715 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (9m) 115s ago 9m 14.4M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:49:53.715 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (4m) 29s ago 16m 501M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:49:53.715 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (4m) 115s ago 14m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:49:53.715 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (3m) 29s ago 16m 45.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:49:53.715 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (3m) 115s ago 14m 34.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (9m) 29s ago 9m 277M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (8m) 115s ago 8m 83.5M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (5m) 29s ago 14m 12.1M - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (5m) 115s ago 14m 21.1M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (87s) 29s ago 13m 254M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (39s) 29s ago 13m 12.1M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (12m) 29s ago 12m 572M 1937M 17.2.0 e1d6a67b021e 1e0cb1d48899 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (12m) 29s ago 12m 395M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (12m) 115s ago 11m 567M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (11m) 115s ago 11m 430M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (11m) 115s ago 11m 331M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (10m) 115s ago 10m 523M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:49:53.716 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (4m) 29s ago 14m 60.9M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:49:54.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 229560 10.49 MB/sec execute 408 sec latency 113.478 ms 2024-02-10T05:49:54.301 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:49:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:49:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:49:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:49:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:49:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:49:54.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:49:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:49:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-10T05:49:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:49:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:49:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:49:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:49:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:49:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:49:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-10T05:49:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 6 2024-02-10T05:49:54.302 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:49:54.302 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:49:54.302 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:54 smithi028 ceph-mon[66336]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:54.303 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:54 smithi028 ceph-mon[66336]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:54.303 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:54 smithi028 ceph-mon[66336]: osd.2 marked itself down and dead 2024-02-10T05:49:54.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:54 smithi192 ceph-mon[52881]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:54.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:54 smithi192 ceph-mon[52881]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:54.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:54 smithi192 ceph-mon[52881]: osd.2 marked itself down and dead 2024-02-10T05:49:54.642 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:49:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:49:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:49:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:49:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:49:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:49:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:49:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:49:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:49:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "8/23 daemons upgraded", 2024-02-10T05:49:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T05:49:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:49:54.644 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:49:54.990 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:54 smithi028 podman[77211]: 2024-02-10 05:49:54.729401156 +0000 UTC m=+1.727999358 container died 1e0cb1d48899368c16c4cc7f2bfd4b5c87afa32820b54f877f5907b776ed0212 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.component=centos-stream-container, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., release=754, ceph=True, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T05:49:55.055 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_WARN 1 osds down 2024-02-10T05:49:55.055 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-10T05:49:55.055 INFO:teuthology.orchestra.run.smithi028.stdout: osd.2 (root=default,host=smithi028) is down 2024-02-10T05:49:55.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 229574 10.46 MB/sec execute 409 sec latency 1068.791 ms 2024-02-10T05:49:55.241 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:55 smithi028 podman[77211]: 2024-02-10 05:49:55.172689041 +0000 UTC m=+2.171287244 container cleanup 1e0cb1d48899368c16c4cc7f2bfd4b5c87afa32820b54f877f5907b776ed0212 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, vendor=Red Hat, Inc., name=centos-stream, ceph=True, io.buildah.version=1.19.8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754) 2024-02-10T05:49:55.241 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:55 smithi028 bash[77211]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2 2024-02-10T05:49:55.241 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:55 smithi028 ceph-mon[66336]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:55.241 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:55 smithi028 ceph-mon[66336]: pgmap v135: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.1 MiB/s wr, 423 op/s 2024-02-10T05:49:55.241 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:55 smithi028 ceph-mon[66336]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:49:55.241 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:55 smithi028 ceph-mon[66336]: failed to encode map e105 with expected crc 2024-02-10T05:49:55.241 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:55 smithi028 ceph-mon[66336]: failed to encode map e105 with expected crc 2024-02-10T05:49:55.241 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:55 smithi028 ceph-mon[66336]: failed to encode map e105 with expected crc 2024-02-10T05:49:55.241 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:55 smithi028 ceph-mon[66336]: failed to encode map e105 with expected crc 2024-02-10T05:49:55.242 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:55 smithi028 ceph-mon[66336]: osdmap e105: 8 total, 7 up, 8 in 2024-02-10T05:49:55.242 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:55 smithi028 ceph-mon[66336]: failed to encode map e105 with expected crc 2024-02-10T05:49:55.242 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:55 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/742080620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:49:55.242 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:55 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/11367376' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:49:55.473 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:55 smithi192 ceph-mon[52881]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:55.473 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:55 smithi192 ceph-mon[52881]: pgmap v135: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.1 MiB/s wr, 423 op/s 2024-02-10T05:49:55.473 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:55 smithi192 ceph-mon[52881]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:49:55.473 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:55 smithi192 ceph-mon[52881]: failed to encode map e105 with expected crc 2024-02-10T05:49:55.473 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:55 smithi192 ceph-mon[52881]: failed to encode map e105 with expected crc 2024-02-10T05:49:55.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:55 smithi192 ceph-mon[52881]: failed to encode map e105 with expected crc 2024-02-10T05:49:55.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:55 smithi192 ceph-mon[52881]: failed to encode map e105 with expected crc 2024-02-10T05:49:55.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:55 smithi192 ceph-mon[52881]: osdmap e105: 8 total, 7 up, 8 in 2024-02-10T05:49:55.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:55 smithi192 ceph-mon[52881]: failed to encode map e105 with expected crc 2024-02-10T05:49:55.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:55 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/742080620' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:49:55.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:55 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/11367376' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:49:55.751 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:55 smithi028 podman[77300]: 2024-02-10 05:49:55.494507235 +0000 UTC m=+0.752717880 container remove 1e0cb1d48899368c16c4cc7f2bfd4b5c87afa32820b54f877f5907b776ed0212 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., name=centos-stream, version=8, io.openshift.expose-services=, GIT_CLEAN=True, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-10T05:49:56.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 229949 10.45 MB/sec execute 410 sec latency 59.624 ms 2024-02-10T05:49:56.259 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:56 smithi028 podman[77405]: 2024-02-10 05:49:55.999753803 +0000 UTC m=+0.028140543 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:49:56.260 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:56 smithi028 podman[77405]: 2024-02-10 05:49:56.199880964 +0000 UTC m=+0.228267709 container create b33ac506ef6ef3604afe4e6aad9231b0a5efb374aa5b09212e743b32f6da0be9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=main-290212a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0) 2024-02-10T05:49:56.260 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:56 smithi028 ceph-mon[66336]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:56.260 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:56 smithi028 ceph-mon[66336]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.260 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:56 smithi028 ceph-mon[66336]: osdmap e106: 8 total, 7 up, 8 in 2024-02-10T05:49:56.260 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:56 smithi028 ceph-mon[66336]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.260 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:56 smithi028 ceph-mon[66336]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.260 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:56 smithi028 ceph-mon[66336]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.260 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:56 smithi028 ceph-mon[66336]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.261 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:56 smithi028 ceph-mon[66336]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.261 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:56 smithi028 ceph-mon[66336]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.261 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:56 smithi028 ceph-mon[66336]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.261 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:56 smithi028 ceph-mon[66336]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:56 smithi192 ceph-mon[52881]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:49:56.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:56 smithi192 ceph-mon[52881]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:56 smithi192 ceph-mon[52881]: osdmap e106: 8 total, 7 up, 8 in 2024-02-10T05:49:56.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:56 smithi192 ceph-mon[52881]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:56 smithi192 ceph-mon[52881]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:56 smithi192 ceph-mon[52881]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:56 smithi192 ceph-mon[52881]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:56 smithi192 ceph-mon[52881]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:56 smithi192 ceph-mon[52881]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:56 smithi192 ceph-mon[52881]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:56 smithi192 ceph-mon[52881]: failed to encode map e106 with expected crc 2024-02-10T05:49:56.765 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:56 smithi028 podman[77405]: 2024-02-10 05:49:56.749074291 +0000 UTC m=+0.777461046 container init b33ac506ef6ef3604afe4e6aad9231b0a5efb374aa5b09212e743b32f6da0be9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-10T05:49:56.766 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:56 smithi028 podman[77405]: 2024-02-10 05:49:56.75675814 +0000 UTC m=+0.785144888 container start b33ac506ef6ef3604afe4e6aad9231b0a5efb374aa5b09212e743b32f6da0be9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-10T05:49:57.018 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:56 smithi028 podman[77405]: 2024-02-10 05:49:56.902889963 +0000 UTC m=+0.931276715 container attach b33ac506ef6ef3604afe4e6aad9231b0a5efb374aa5b09212e743b32f6da0be9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True) 2024-02-10T05:49:57.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 230411 10.46 MB/sec execute 411 sec latency 30.326 ms 2024-02-10T05:49:57.268 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:57 smithi028 ceph-mon[66336]: pgmap v138: 97 pgs: 2 activating+undersized, 8 active+undersized+degraded, 4 stale+active+clean, 2 active+undersized, 81 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.6 KiB/s rd, 3.4 MiB/s wr, 304 op/s; 69/1545 objects degraded (4.466%) 2024-02-10T05:49:57.269 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:57 smithi028 ceph-mon[66336]: Health check failed: Degraded data redundancy: 69/1545 objects degraded (4.466%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:57.269 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:57 smithi028 podman[77405]: 2024-02-10 05:49:57.229360494 +0000 UTC m=+1.257747239 container died b33ac506ef6ef3604afe4e6aad9231b0a5efb374aa5b09212e743b32f6da0be9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-10T05:49:57.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:57 smithi192 ceph-mon[52881]: pgmap v138: 97 pgs: 2 activating+undersized, 8 active+undersized+degraded, 4 stale+active+clean, 2 active+undersized, 81 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.6 KiB/s rd, 3.4 MiB/s wr, 304 op/s; 69/1545 objects degraded (4.466%) 2024-02-10T05:49:57.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:57 smithi192 ceph-mon[52881]: Health check failed: Degraded data redundancy: 69/1545 objects degraded (4.466%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T05:49:58.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 230889 10.48 MB/sec execute 412 sec latency 29.754 ms 2024-02-10T05:49:58.293 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:58 smithi028 podman[77431]: 2024-02-10 05:49:58.275724479 +0000 UTC m=+1.039035302 container remove b33ac506ef6ef3604afe4e6aad9231b0a5efb374aa5b09212e743b32f6da0be9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-10T05:49:58.545 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:58 smithi028 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-10T05:49:58.545 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:58 smithi028 systemd[1]: Stopped Ceph osd.2 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:49:58.545 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:58 smithi028 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.2.service: Consumed 3min 17.226s CPU time. 2024-02-10T05:49:58.800 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:58 smithi028 systemd[1]: Starting Ceph osd.2 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:49:59.052 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:49:58 smithi028 ceph-mon[66336]: pgmap v139: 97 pgs: 2 activating+undersized, 15 active+undersized+degraded, 2 stale+active+clean, 3 active+undersized, 75 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.6 MiB/s wr, 425 op/s; 101/1482 objects degraded (6.815%) 2024-02-10T05:49:59.052 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:58 smithi028 podman[77523]: 2024-02-10 05:49:58.829338107 +0000 UTC m=+0.025106362 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:49:59.052 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:58 smithi028 podman[77523]: 2024-02-10 05:49:58.935120324 +0000 UTC m=+0.130888618 container create e180014a6d4ca508840c60966496244b8a0243b97ba7ffc6a5cc9ee92a787988 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-10T05:49:59.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 231228 10.49 MB/sec execute 413 sec latency 46.501 ms 2024-02-10T05:49:59.303 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 podman[77523]: 2024-02-10 05:49:59.204710459 +0000 UTC m=+0.400478709 container init e180014a6d4ca508840c60966496244b8a0243b97ba7ffc6a5cc9ee92a787988 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-02-10T05:49:59.304 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 podman[77523]: 2024-02-10 05:49:59.211147198 +0000 UTC m=+0.406915445 container start e180014a6d4ca508840c60966496244b8a0243b97ba7ffc6a5cc9ee92a787988 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-10T05:49:59.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:49:58 smithi192 ceph-mon[52881]: pgmap v139: 97 pgs: 2 activating+undersized, 15 active+undersized+degraded, 2 stale+active+clean, 3 active+undersized, 75 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.6 MiB/s wr, 425 op/s; 101/1482 objects degraded (6.815%) 2024-02-10T05:49:59.569 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 podman[77523]: 2024-02-10 05:49:59.33237905 +0000 UTC m=+0.528147294 container attach e180014a6d4ca508840c60966496244b8a0243b97ba7ffc6a5cc9ee92a787988 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-10T05:50:00.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 101/1482 objects degraded (6.815%), 15 pgs degraded 2024-02-10T05:50:00.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: [WRN] OSD_DOWN: 1 osds down 2024-02-10T05:50:00.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: osd.2 (root=default,host=smithi028) is down 2024-02-10T05:50:00.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: [WRN] PG_DEGRADED: Degraded data redundancy: 101/1482 objects degraded (6.815%), 15 pgs degraded 2024-02-10T05:50:00.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-02-10T05:50:00.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-02-10T05:50:00.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 2.7 is active+undersized+degraded, acting [6,7] 2024-02-10T05:50:00.072 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 2.c is active+undersized+degraded, acting [5,0] 2024-02-10T05:50:00.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 2.e is active+undersized+degraded, acting [3,7] 2024-02-10T05:50:00.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 2.16 is active+undersized+degraded, acting [5,6] 2024-02-10T05:50:00.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-02-10T05:50:00.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 2.1c is active+undersized+degraded, acting [7,5] 2024-02-10T05:50:00.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-02-10T05:50:00.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-02-10T05:50:00.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-02-10T05:50:00.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-02-10T05:50:00.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-02-10T05:50:00.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-02-10T05:50:00.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: pg 4.19 is active+undersized+degraded, acting [3,0] 2024-02-10T05:50:00.073 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:00.074 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate[77539]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T05:50:00.074 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate[77539]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T05:50:00.074 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 bash[77523]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T05:50:00.074 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 bash[77523]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T05:50:00.074 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 bash[77523]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T05:50:00.074 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 bash[77523]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T05:50:00.074 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 bash[77523]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-10T05:50:00.074 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 bash[77523]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T05:50:00.074 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 bash[77523]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-10T05:50:00.074 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate[77539]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T05:50:00.074 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate[77539]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T05:50:00.074 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate[77539]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-10T05:50:00.075 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate[77539]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-10T05:50:00.075 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate[77539]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-10T05:50:00.075 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:49:59 smithi028 podman[77523]: 2024-02-10 05:49:59.922176307 +0000 UTC m=+1.117944553 container died e180014a6d4ca508840c60966496244b8a0243b97ba7ffc6a5cc9ee92a787988 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.build-date=20240131) 2024-02-10T05:50:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 231535 10.49 MB/sec execute 414 sec latency 51.685 ms 2024-02-10T05:50:00.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 101/1482 objects degraded (6.815%), 15 pgs degraded 2024-02-10T05:50:00.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: [WRN] OSD_DOWN: 1 osds down 2024-02-10T05:50:00.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: osd.2 (root=default,host=smithi028) is down 2024-02-10T05:50:00.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: [WRN] PG_DEGRADED: Degraded data redundancy: 101/1482 objects degraded (6.815%), 15 pgs degraded 2024-02-10T05:50:00.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-02-10T05:50:00.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-02-10T05:50:00.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 2.7 is active+undersized+degraded, acting [6,7] 2024-02-10T05:50:00.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 2.c is active+undersized+degraded, acting [5,0] 2024-02-10T05:50:00.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 2.e is active+undersized+degraded, acting [3,7] 2024-02-10T05:50:00.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 2.16 is active+undersized+degraded, acting [5,6] 2024-02-10T05:50:00.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-02-10T05:50:00.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 2.1c is active+undersized+degraded, acting [7,5] 2024-02-10T05:50:00.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-02-10T05:50:00.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-02-10T05:50:00.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-02-10T05:50:00.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-02-10T05:50:00.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-02-10T05:50:00.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-02-10T05:50:00.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: pg 4.19 is active+undersized+degraded, acting [3,0] 2024-02-10T05:50:00.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:01.094 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:01 smithi028 ceph-mon[66336]: pgmap v140: 97 pgs: 2 activating+undersized, 19 active+undersized+degraded, 5 active+undersized, 71 active+clean; 334 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 MiB/s wr, 302 op/s; 142/1506 objects degraded (9.429%) 2024-02-10T05:50:01.094 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:50:01.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 231834 10.48 MB/sec execute 415 sec latency 55.026 ms 2024-02-10T05:50:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:01 smithi192 ceph-mon[52881]: pgmap v140: 97 pgs: 2 activating+undersized, 19 active+undersized+degraded, 5 active+undersized, 71 active+clean; 334 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 MiB/s wr, 302 op/s; 142/1506 objects degraded (9.429%) 2024-02-10T05:50:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:50:01.603 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:50:01 smithi028 podman[77671]: 2024-02-10 05:50:01.546751631 +0000 UTC m=+1.616289873 container remove e180014a6d4ca508840c60966496244b8a0243b97ba7ffc6a5cc9ee92a787988 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-activate, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_CLEAN=True, RELEASE=main-290212a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:50:01.857 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:50:01 smithi028 podman[77724]: 2024-02-10 05:50:01.748113627 +0000 UTC m=+0.033949220 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:50:02.117 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:02 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 157/1707 objects degraded (9.197%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:02.117 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:50:01 smithi028 podman[77724]: 2024-02-10 05:50:01.864425399 +0000 UTC m=+0.150260981 container create a45f0e24ae449906f47b8a167169e276435a0739e458c6ec2206d8b057332d03 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:50:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 232088 10.46 MB/sec execute 416 sec latency 47.073 ms 2024-02-10T05:50:02.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:02 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 157/1707 objects degraded (9.197%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:02.621 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:50:02 smithi028 podman[77724]: 2024-02-10 05:50:02.418467879 +0000 UTC m=+0.704303454 container init a45f0e24ae449906f47b8a167169e276435a0739e458c6ec2206d8b057332d03 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2) 2024-02-10T05:50:02.622 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:50:02 smithi028 podman[77724]: 2024-02-10 05:50:02.426984712 +0000 UTC m=+0.712820290 container start a45f0e24ae449906f47b8a167169e276435a0739e458c6ec2206d8b057332d03 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-10T05:50:02.622 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:50:02 smithi028 ceph-osd[77740]: -- 172.21.15.28:0/1834123602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555df731f1e0 con 0x555df72f1000 2024-02-10T05:50:02.622 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:50:02 smithi028 bash[77724]: a45f0e24ae449906f47b8a167169e276435a0739e458c6ec2206d8b057332d03 2024-02-10T05:50:02.622 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:50:02 smithi028 systemd[1]: Started Ceph osd.2 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:50:03.139 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:03 smithi028 ceph-mon[66336]: pgmap v141: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 395 op/s; 157/1707 objects degraded (9.197%) 2024-02-10T05:50:03.140 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:03 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:03.140 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:03 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:03.140 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:03 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:03.140 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:50:03 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2[77736]: 2024-02-10T05:50:02.999+0000 7f057ed1e7c0 -1 Falling back to public interface 2024-02-10T05:50:03.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 232450 10.46 MB/sec execute 417 sec latency 58.233 ms 2024-02-10T05:50:03.575 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:03 smithi192 ceph-mon[52881]: pgmap v141: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 345 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 395 op/s; 157/1707 objects degraded (9.197%) 2024-02-10T05:50:03.575 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:03 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:03.575 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:03 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:03.575 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:03 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:04.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 232812 10.45 MB/sec execute 418 sec latency 77.611 ms 2024-02-10T05:50:05.138 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:05 smithi028 ceph-mon[66336]: pgmap v142: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.6 MiB/s wr, 452 op/s; 154/1698 objects degraded (9.069%) 2024-02-10T05:50:05.170 INFO:teuthology.orchestra.run.smithi028.stdout: 5 233161 10.44 MB/sec execute 419 sec latency 47.078 ms 2024-02-10T05:50:05.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:05 smithi192 ceph-mon[52881]: pgmap v142: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 346 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.6 MiB/s wr, 452 op/s; 154/1698 objects degraded (9.069%) 2024-02-10T05:50:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 233503 10.43 MB/sec execute 420 sec latency 57.912 ms 2024-02-10T05:50:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 233921 10.44 MB/sec execute 421 sec latency 46.290 ms 2024-02-10T05:50:07.255 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:07 smithi028 ceph-mon[66336]: pgmap v143: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 5.5 MiB/s wr, 594 op/s; 149/1632 objects degraded (9.130%) 2024-02-10T05:50:07.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:07 smithi192 ceph-mon[52881]: pgmap v143: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 336 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 5.5 MiB/s wr, 594 op/s; 149/1632 objects degraded (9.130%) 2024-02-10T05:50:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 234365 10.46 MB/sec execute 422 sec latency 30.815 ms 2024-02-10T05:50:08.270 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:08 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 149/1632 objects degraded (9.130%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:08.424 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:50:08 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2[77736]: 2024-02-10T05:50:08.379+0000 7f057ed1e7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-02-10T05:50:08.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:08 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 149/1632 objects degraded (9.130%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:09.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 234781 10.47 MB/sec execute 423 sec latency 41.164 ms 2024-02-10T05:50:09.287 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:09 smithi028 ceph-mon[66336]: pgmap v144: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 334 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.8 MiB/s wr, 530 op/s; 148/1602 objects degraded (9.238%) 2024-02-10T05:50:09.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:09 smithi192 ceph-mon[52881]: pgmap v144: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 334 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.8 MiB/s wr, 530 op/s; 148/1602 objects degraded (9.238%) 2024-02-10T05:50:09.792 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:50:09 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2[77736]: 2024-02-10T05:50:09.611+0000 7f057ed1e7c0 -1 osd.2 104 log_to_monitors true 2024-02-10T05:50:10.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 235190 10.47 MB/sec execute 424 sec latency 27.775 ms 2024-02-10T05:50:10.299 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:10 smithi028 ceph-mon[66336]: from='osd.2 [v2:172.21.15.28:6818/3604919958,v1:172.21.15.28:6819/3604919958]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:50:10.299 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:50:10 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2[77736]: 2024-02-10T05:50:10.152+0000 7f0570176700 -1 osd.2 104 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:50:10.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:10 smithi192 ceph-mon[52881]: from='osd.2 [v2:172.21.15.28:6818/3604919958,v1:172.21.15.28:6819/3604919958]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:50:11.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 235518 10.46 MB/sec execute 425 sec latency 52.809 ms 2024-02-10T05:50:11.549 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:11 smithi028 ceph-mon[66336]: pgmap v145: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 334 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.7 MiB/s wr, 453 op/s; 148/1596 objects degraded (9.273%) 2024-02-10T05:50:11.549 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:11 smithi028 ceph-mon[66336]: from='osd.2 [v2:172.21.15.28:6818/3604919958,v1:172.21.15.28:6819/3604919958]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:50:11.550 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:11 smithi028 ceph-mon[66336]: failed to encode map e107 with expected crc 2024-02-10T05:50:11.550 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:11 smithi028 ceph-mon[66336]: osdmap e107: 8 total, 7 up, 8 in 2024-02-10T05:50:11.550 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:11 smithi028 ceph-mon[66336]: from='osd.2 [v2:172.21.15.28:6818/3604919958,v1:172.21.15.28:6819/3604919958]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:50:11.550 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:11 smithi028 ceph-mon[66336]: failed to encode map e107 with expected crc 2024-02-10T05:50:11.550 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:11 smithi028 ceph-mon[66336]: failed to encode map e107 with expected crc 2024-02-10T05:50:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:11 smithi192 ceph-mon[52881]: pgmap v145: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 334 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.7 MiB/s wr, 453 op/s; 148/1596 objects degraded (9.273%) 2024-02-10T05:50:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:11 smithi192 ceph-mon[52881]: from='osd.2 [v2:172.21.15.28:6818/3604919958,v1:172.21.15.28:6819/3604919958]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:50:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:11 smithi192 ceph-mon[52881]: failed to encode map e107 with expected crc 2024-02-10T05:50:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:11 smithi192 ceph-mon[52881]: osdmap e107: 8 total, 7 up, 8 in 2024-02-10T05:50:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:11 smithi192 ceph-mon[52881]: from='osd.2 [v2:172.21.15.28:6818/3604919958,v1:172.21.15.28:6819/3604919958]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:50:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:11 smithi192 ceph-mon[52881]: failed to encode map e107 with expected crc 2024-02-10T05:50:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:11 smithi192 ceph-mon[52881]: failed to encode map e107 with expected crc 2024-02-10T05:50:12.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 235518 10.44 MB/sec execute 426 sec latency 1052.840 ms 2024-02-10T05:50:12.305 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:12 smithi028 ceph-mon[66336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:50:12.305 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:12 smithi028 ceph-mon[66336]: osd.2 [v2:172.21.15.28:6818/3604919958,v1:172.21.15.28:6819/3604919958] boot 2024-02-10T05:50:12.305 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:12 smithi028 ceph-mon[66336]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T05:50:12.305 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:12 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:50:12.305 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:12 smithi028 ceph-mon[66336]: failed to encode map e108 with expected crc 2024-02-10T05:50:12.305 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:12 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:12.305 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:12 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:12.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:12 smithi192 ceph-mon[52881]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:50:12.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:12 smithi192 ceph-mon[52881]: osd.2 [v2:172.21.15.28:6818/3604919958,v1:172.21.15.28:6819/3604919958] boot 2024-02-10T05:50:12.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:12 smithi192 ceph-mon[52881]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T05:50:12.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:12 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:50:12.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:12 smithi192 ceph-mon[52881]: failed to encode map e108 with expected crc 2024-02-10T05:50:12.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:12 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:12.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:12 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:13.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 235788 10.42 MB/sec execute 427 sec latency 1117.104 ms 2024-02-10T05:50:13.319 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:13 smithi028 ceph-mon[66336]: pgmap v148: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.4 MiB/s wr, 641 op/s; 155/1731 objects degraded (8.954%) 2024-02-10T05:50:13.320 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:13 smithi028 ceph-mon[66336]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.320 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:13 smithi028 ceph-mon[66336]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T05:50:13.320 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:13 smithi028 ceph-mon[66336]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.320 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:13 smithi028 ceph-mon[66336]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.320 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:13 smithi028 ceph-mon[66336]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.320 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:13 smithi028 ceph-mon[66336]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.320 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:13 smithi028 ceph-mon[66336]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.320 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:13 smithi028 ceph-mon[66336]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.320 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:13 smithi028 ceph-mon[66336]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.320 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:13 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 155/1731 objects degraded (8.954%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:13 smithi192 ceph-mon[52881]: pgmap v148: 97 pgs: 20 active+undersized+degraded, 6 active+undersized, 71 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.4 MiB/s wr, 641 op/s; 155/1731 objects degraded (8.954%) 2024-02-10T05:50:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:13 smithi192 ceph-mon[52881]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:13 smithi192 ceph-mon[52881]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T05:50:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:13 smithi192 ceph-mon[52881]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:13 smithi192 ceph-mon[52881]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:13 smithi192 ceph-mon[52881]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:13 smithi192 ceph-mon[52881]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:13 smithi192 ceph-mon[52881]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:13 smithi192 ceph-mon[52881]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:13 smithi192 ceph-mon[52881]: failed to encode map e109 with expected crc 2024-02-10T05:50:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:13 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 155/1731 objects degraded (8.954%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:14.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 235990 10.41 MB/sec execute 428 sec latency 998.396 ms 2024-02-10T05:50:14.338 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:14 smithi028 ceph-mon[66336]: failed to encode map e110 with expected crc 2024-02-10T05:50:14.339 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:14 smithi028 ceph-mon[66336]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T05:50:14.339 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:14 smithi028 ceph-mon[66336]: failed to encode map e110 with expected crc 2024-02-10T05:50:14.339 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:14 smithi028 ceph-mon[66336]: failed to encode map e110 with expected crc 2024-02-10T05:50:14.339 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:14 smithi028 ceph-mon[66336]: failed to encode map e110 with expected crc 2024-02-10T05:50:14.339 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:14 smithi028 ceph-mon[66336]: failed to encode map e110 with expected crc 2024-02-10T05:50:14.339 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:14 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:14.339 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:14 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:14 smithi192 ceph-mon[52881]: failed to encode map e110 with expected crc 2024-02-10T05:50:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:14 smithi192 ceph-mon[52881]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T05:50:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:14 smithi192 ceph-mon[52881]: failed to encode map e110 with expected crc 2024-02-10T05:50:14.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:14 smithi192 ceph-mon[52881]: failed to encode map e110 with expected crc 2024-02-10T05:50:14.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:14 smithi192 ceph-mon[52881]: failed to encode map e110 with expected crc 2024-02-10T05:50:14.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:14 smithi192 ceph-mon[52881]: failed to encode map e110 with expected crc 2024-02-10T05:50:14.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:14 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:14.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:14 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:15.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 236324 10.41 MB/sec execute 429 sec latency 590.941 ms 2024-02-10T05:50:15.356 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:15 smithi028 ceph-mon[66336]: pgmap v151: 97 pgs: 2 remapped+peering, 17 active+undersized+degraded, 5 active+undersized, 73 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 8.1 MiB/s wr, 706 op/s; 123/1740 objects degraded (7.069%) 2024-02-10T05:50:15.356 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:15 smithi028 ceph-mon[66336]: failed to encode map e111 with expected crc 2024-02-10T05:50:15.356 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:15 smithi028 ceph-mon[66336]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T05:50:15.356 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:15 smithi028 ceph-mon[66336]: failed to encode map e111 with expected crc 2024-02-10T05:50:15.356 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:15 smithi028 ceph-mon[66336]: failed to encode map e111 with expected crc 2024-02-10T05:50:15.356 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:15 smithi028 ceph-mon[66336]: failed to encode map e111 with expected crc 2024-02-10T05:50:15.356 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:15 smithi028 ceph-mon[66336]: failed to encode map e111 with expected crc 2024-02-10T05:50:15.357 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:15.357 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:50:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:15 smithi192 ceph-mon[52881]: pgmap v151: 97 pgs: 2 remapped+peering, 17 active+undersized+degraded, 5 active+undersized, 73 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 8.1 MiB/s wr, 706 op/s; 123/1740 objects degraded (7.069%) 2024-02-10T05:50:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:15 smithi192 ceph-mon[52881]: failed to encode map e111 with expected crc 2024-02-10T05:50:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:15 smithi192 ceph-mon[52881]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T05:50:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:15 smithi192 ceph-mon[52881]: failed to encode map e111 with expected crc 2024-02-10T05:50:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:15 smithi192 ceph-mon[52881]: failed to encode map e111 with expected crc 2024-02-10T05:50:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:15 smithi192 ceph-mon[52881]: failed to encode map e111 with expected crc 2024-02-10T05:50:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:15 smithi192 ceph-mon[52881]: failed to encode map e111 with expected crc 2024-02-10T05:50:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:50:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 236738 10.40 MB/sec execute 430 sec latency 854.533 ms 2024-02-10T05:50:16.366 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:16 smithi028 ceph-mon[66336]: osdmap e112: 8 total, 8 up, 8 in 2024-02-10T05:50:16.366 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:16 smithi028 ceph-mon[66336]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.367 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:16 smithi028 ceph-mon[66336]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.367 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:16 smithi028 ceph-mon[66336]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.367 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:16 smithi028 ceph-mon[66336]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.367 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:16 smithi028 ceph-mon[66336]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.367 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:16 smithi028 ceph-mon[66336]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.367 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:16 smithi028 ceph-mon[66336]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.367 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:16 smithi028 ceph-mon[66336]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:16 smithi192 ceph-mon[52881]: osdmap e112: 8 total, 8 up, 8 in 2024-02-10T05:50:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:16 smithi192 ceph-mon[52881]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:16 smithi192 ceph-mon[52881]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:16 smithi192 ceph-mon[52881]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:16 smithi192 ceph-mon[52881]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:16 smithi192 ceph-mon[52881]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:16 smithi192 ceph-mon[52881]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:16 smithi192 ceph-mon[52881]: failed to encode map e112 with expected crc 2024-02-10T05:50:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:16 smithi192 ceph-mon[52881]: failed to encode map e112 with expected crc 2024-02-10T05:50:17.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 237010 10.38 MB/sec execute 431 sec latency 916.251 ms 2024-02-10T05:50:17.454 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:17 smithi028 ceph-mon[66336]: pgmap v154: 97 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 peering, 3 active+undersized+degraded, 1 active+undersized, 87 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.1 MiB/s wr, 367 op/s; 14/1635 objects degraded (0.856%); 2.3 MiB/s, 27 objects/s recovering 2024-02-10T05:50:17.455 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:17 smithi028 ceph-mon[66336]: failed to encode map e113 with expected crc 2024-02-10T05:50:17.455 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:17 smithi028 ceph-mon[66336]: osdmap e113: 8 total, 8 up, 8 in 2024-02-10T05:50:17.455 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:17 smithi028 ceph-mon[66336]: failed to encode map e113 with expected crc 2024-02-10T05:50:17.455 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:17 smithi028 ceph-mon[66336]: failed to encode map e113 with expected crc 2024-02-10T05:50:17.455 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:17 smithi028 ceph-mon[66336]: failed to encode map e113 with expected crc 2024-02-10T05:50:17.455 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:17 smithi028 ceph-mon[66336]: failed to encode map e113 with expected crc 2024-02-10T05:50:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:17 smithi192 ceph-mon[52881]: pgmap v154: 97 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 peering, 3 active+undersized+degraded, 1 active+undersized, 87 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.1 MiB/s wr, 367 op/s; 14/1635 objects degraded (0.856%); 2.3 MiB/s, 27 objects/s recovering 2024-02-10T05:50:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:17 smithi192 ceph-mon[52881]: failed to encode map e113 with expected crc 2024-02-10T05:50:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:17 smithi192 ceph-mon[52881]: osdmap e113: 8 total, 8 up, 8 in 2024-02-10T05:50:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:17 smithi192 ceph-mon[52881]: failed to encode map e113 with expected crc 2024-02-10T05:50:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:17 smithi192 ceph-mon[52881]: failed to encode map e113 with expected crc 2024-02-10T05:50:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:17 smithi192 ceph-mon[52881]: failed to encode map e113 with expected crc 2024-02-10T05:50:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:17 smithi192 ceph-mon[52881]: failed to encode map e113 with expected crc 2024-02-10T05:50:18.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 237303 10.37 MB/sec execute 432 sec latency 971.881 ms 2024-02-10T05:50:18.212 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:18 smithi028 ceph-mon[66336]: failed to encode map e114 with expected crc 2024-02-10T05:50:18.213 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:18 smithi028 ceph-mon[66336]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T05:50:18.213 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:18 smithi028 ceph-mon[66336]: failed to encode map e114 with expected crc 2024-02-10T05:50:18.213 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:18 smithi028 ceph-mon[66336]: failed to encode map e114 with expected crc 2024-02-10T05:50:18.213 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:18 smithi028 ceph-mon[66336]: failed to encode map e114 with expected crc 2024-02-10T05:50:18.213 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:18 smithi028 ceph-mon[66336]: failed to encode map e114 with expected crc 2024-02-10T05:50:18.213 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:18 smithi028 ceph-mon[66336]: failed to encode map e114 with expected crc 2024-02-10T05:50:18.213 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:18 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 14/1635 objects degraded (0.856%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:18.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:18 smithi192 ceph-mon[52881]: failed to encode map e114 with expected crc 2024-02-10T05:50:18.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:18 smithi192 ceph-mon[52881]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T05:50:18.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:18 smithi192 ceph-mon[52881]: failed to encode map e114 with expected crc 2024-02-10T05:50:18.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:18 smithi192 ceph-mon[52881]: failed to encode map e114 with expected crc 2024-02-10T05:50:18.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:18 smithi192 ceph-mon[52881]: failed to encode map e114 with expected crc 2024-02-10T05:50:18.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:18 smithi192 ceph-mon[52881]: failed to encode map e114 with expected crc 2024-02-10T05:50:18.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:18 smithi192 ceph-mon[52881]: failed to encode map e114 with expected crc 2024-02-10T05:50:18.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:18 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 14/1635 objects degraded (0.856%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:19.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 237689 10.39 MB/sec execute 433 sec latency 43.990 ms 2024-02-10T05:50:19.228 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:19 smithi028 ceph-mon[66336]: pgmap v157: 97 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 peering, 91 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s wr, 383 op/s; 2.3 MiB/s, 31 objects/s recovering 2024-02-10T05:50:19.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:19 smithi028 ceph-mon[66336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/1635 objects degraded (0.856%), 3 pgs degraded) 2024-02-10T05:50:19.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:19 smithi028 ceph-mon[66336]: Cluster is now healthy 2024-02-10T05:50:19.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:19 smithi192 ceph-mon[52881]: pgmap v157: 97 pgs: 1 active+undersized+remapped, 2 remapped+peering, 3 peering, 91 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.0 MiB/s wr, 383 op/s; 2.3 MiB/s, 31 objects/s recovering 2024-02-10T05:50:19.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:19 smithi192 ceph-mon[52881]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/1635 objects degraded (0.856%), 3 pgs degraded) 2024-02-10T05:50:19.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:19 smithi192 ceph-mon[52881]: Cluster is now healthy 2024-02-10T05:50:20.171 INFO:teuthology.orchestra.run.smithi028.stdout: 5 238119 10.41 MB/sec execute 434 sec latency 43.090 ms 2024-02-10T05:50:20.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:20.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:20.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:50:20.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:50:20.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:20.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:20.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:20.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:50:20.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:50:20.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:50:20.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T05:50:20.501 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T05:50:20.501 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-02-10T05:50:20.501 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:20 smithi028 ceph-mon[66336]: pgmap v158: 97 pgs: 1 active+undersized+remapped, 3 peering, 93 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s wr, 416 op/s; 1.8 MiB/s, 29 objects/s recovering 2024-02-10T05:50:20.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:20.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:20.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:50:20.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:50:20.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:20.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:20.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:20.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:50:20.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:50:20.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:50:20.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T05:50:20.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T05:50:20.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-02-10T05:50:20.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:20 smithi192 ceph-mon[52881]: pgmap v158: 97 pgs: 1 active+undersized+remapped, 3 peering, 93 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.8 MiB/s wr, 416 op/s; 1.8 MiB/s, 29 objects/s recovering 2024-02-10T05:50:21.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 238493 10.42 MB/sec execute 435 sec latency 36.606 ms 2024-02-10T05:50:22.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 238738 10.41 MB/sec execute 436 sec latency 39.154 ms 2024-02-10T05:50:23.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 239006 10.39 MB/sec execute 437 sec latency 47.367 ms 2024-02-10T05:50:23.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:23 smithi028 ceph-mon[66336]: pgmap v159: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.8 MiB/s wr, 402 op/s; 1.3 MiB/s, 21 objects/s recovering 2024-02-10T05:50:23.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:23 smithi192 ceph-mon[52881]: pgmap v159: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.8 MiB/s wr, 402 op/s; 1.3 MiB/s, 21 objects/s recovering 2024-02-10T05:50:24.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 239252 10.37 MB/sec execute 438 sec latency 96.494 ms 2024-02-10T05:50:25.095 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:25 smithi028 ceph-mon[66336]: pgmap v160: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.2 MiB/s wr, 387 op/s; 147 KiB/s, 7 objects/s recovering 2024-02-10T05:50:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 239473 10.36 MB/sec execute 439 sec latency 69.617 ms 2024-02-10T05:50:25.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:25 smithi192 ceph-mon[52881]: pgmap v160: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.2 MiB/s wr, 387 op/s; 147 KiB/s, 7 objects/s recovering 2024-02-10T05:50:25.568 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:50:26.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 239695 10.35 MB/sec execute 440 sec latency 74.023 ms 2024-02-10T05:50:26.457 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:50:26.457 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (6m) 14s ago 15m 15.3M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:50:26.457 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (2m) 14s ago 15m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:50:26.457 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (2m) 2m ago 15m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:50:26.457 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (6m) 14s ago 15m 48.1M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:50:26.457 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (9m) 14s ago 9m 38.0M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:50:26.457 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (9m) 2m ago 9m 14.4M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:50:26.457 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (5m) 14s ago 16m 503M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:50:26.457 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (4m) 2m ago 15m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:50:26.457 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (4m) 14s ago 17m 49.1M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:50:26.457 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (3m) 2m ago 15m 34.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:50:26.457 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (9m) 14s ago 9m 279M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:50:26.458 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (9m) 2m ago 9m 83.5M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:50:26.458 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (6m) 14s ago 15m 11.7M - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:50:26.458 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (5m) 2m ago 14m 21.1M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:50:26.458 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (2m) 14s ago 14m 313M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:50:26.458 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (72s) 14s ago 13m 253M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:50:26.458 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (24s) 14s ago 13m 11.6M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 a45f0e24ae44 2024-02-10T05:50:26.458 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (12m) 14s ago 12m 418M 1937M 17.2.0 e1d6a67b021e 3ea0a9621c0c 2024-02-10T05:50:26.458 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (12m) 2m ago 12m 567M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:50:26.458 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (12m) 2m ago 12m 430M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:50:26.458 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (11m) 2m ago 11m 331M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:50:26.458 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (11m) 2m ago 11m 523M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:50:26.458 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (4m) 14s ago 14m 61.9M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:50:26.937 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:50:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:50:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:50:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:50:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:50:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:50:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:50:26.937 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:50:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-10T05:50:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T05:50:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:50:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:50:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:50:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:50:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:50:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-10T05:50:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 7 2024-02-10T05:50:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:50:26.938 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:50:27.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 240004 10.35 MB/sec execute 441 sec latency 87.900 ms 2024-02-10T05:50:27.175 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:27 smithi028 ceph-mon[66336]: from='client.44239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:27.175 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:27 smithi028 ceph-mon[66336]: pgmap v161: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.4 MiB/s wr, 359 op/s; 120 KiB/s, 4 objects/s recovering 2024-02-10T05:50:27.175 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:27 smithi028 ceph-mon[66336]: from='client.44243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:27.175 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:27 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/1372407074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:50:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:27 smithi192 ceph-mon[52881]: from='client.44239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:27 smithi192 ceph-mon[52881]: pgmap v161: 97 pgs: 97 active+clean; 331 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.4 MiB/s wr, 359 op/s; 120 KiB/s, 4 objects/s recovering 2024-02-10T05:50:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:27 smithi192 ceph-mon[52881]: from='client.44243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:27 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/1372407074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:50:27.341 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:50:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:50:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:50:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:50:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:50:27.341 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:50:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:50:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:50:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:50:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "9/23 daemons upgraded", 2024-02-10T05:50:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T05:50:27.342 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:50:27.342 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:50:27.848 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:50:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 240311 10.33 MB/sec execute 442 sec latency 85.024 ms 2024-02-10T05:50:28.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:28 smithi028 ceph-mon[66336]: from='client.34308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:28.191 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:28 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2894751926' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:50:28.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:28 smithi192 ceph-mon[52881]: from='client.34308 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:28.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:28 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2894751926' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:50:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 240653 10.32 MB/sec execute 443 sec latency 60.504 ms 2024-02-10T05:50:29.205 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:29 smithi028 ceph-mon[66336]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:29.205 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:29 smithi028 ceph-mon[66336]: pgmap v162: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.4 MiB/s wr, 385 op/s; 109 KiB/s, 4 objects/s recovering 2024-02-10T05:50:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:29 smithi192 ceph-mon[52881]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:29 smithi192 ceph-mon[52881]: pgmap v162: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.4 MiB/s wr, 385 op/s; 109 KiB/s, 4 objects/s recovering 2024-02-10T05:50:30.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 240857 10.31 MB/sec execute 444 sec latency 55.392 ms 2024-02-10T05:50:31.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 241119 10.31 MB/sec execute 445 sec latency 63.893 ms 2024-02-10T05:50:31.264 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:31 smithi028 ceph-mon[66336]: pgmap v163: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.3 MiB/s wr, 417 op/s; 98 KiB/s, 3 objects/s recovering 2024-02-10T05:50:31.265 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:31.265 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:50:31.265 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:31 smithi192 ceph-mon[52881]: pgmap v163: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.3 MiB/s wr, 417 op/s; 98 KiB/s, 3 objects/s recovering 2024-02-10T05:50:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:50:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:32.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 241482 10.32 MB/sec execute 446 sec latency 43.918 ms 2024-02-10T05:50:33.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 241870 10.34 MB/sec execute 447 sec latency 54.692 ms 2024-02-10T05:50:33.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:33 smithi028 ceph-mon[66336]: pgmap v164: 97 pgs: 97 active+clean; 329 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.1 MiB/s wr, 498 op/s 2024-02-10T05:50:33.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:33 smithi192 ceph-mon[52881]: pgmap v164: 97 pgs: 97 active+clean; 329 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.1 MiB/s wr, 498 op/s 2024-02-10T05:50:34.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 242114 10.34 MB/sec execute 448 sec latency 101.702 ms 2024-02-10T05:50:35.103 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:35 smithi028 ceph-mon[66336]: pgmap v165: 97 pgs: 97 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 3.6 MiB/s wr, 440 op/s 2024-02-10T05:50:35.103 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:35 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T05:50:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 242312 10.32 MB/sec execute 449 sec latency 42.738 ms 2024-02-10T05:50:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:35 smithi192 ceph-mon[52881]: pgmap v165: 97 pgs: 97 active+clean; 328 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 3.6 MiB/s wr, 440 op/s 2024-02-10T05:50:35.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:35 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T05:50:36.110 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:36 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T05:50:36.110 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:36 smithi028 ceph-mon[66336]: Upgrade: osd.3 is safe to restart 2024-02-10T05:50:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 242709 10.31 MB/sec execute 450 sec latency 29.805 ms 2024-02-10T05:50:36.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:36 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-10T05:50:36.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:36 smithi192 ceph-mon[52881]: Upgrade: osd.3 is safe to restart 2024-02-10T05:50:37.128 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:37 smithi028 ceph-mon[66336]: pgmap v166: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.5 MiB/s wr, 458 op/s 2024-02-10T05:50:37.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 243051 10.30 MB/sec execute 451 sec latency 34.954 ms 2024-02-10T05:50:37.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:37 smithi192 ceph-mon[52881]: pgmap v166: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.5 MiB/s wr, 458 op/s 2024-02-10T05:50:38.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 243331 10.30 MB/sec execute 452 sec latency 111.016 ms 2024-02-10T05:50:38.406 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:38 smithi028 ceph-mon[66336]: Upgrade: Updating osd.3 2024-02-10T05:50:38.407 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:38 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:38.407 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:38 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:50:38.407 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:38 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:50:38.407 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:38 smithi028 ceph-mon[66336]: Deploying daemon osd.3 on smithi028 2024-02-10T05:50:38.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:38 smithi192 ceph-mon[52881]: Upgrade: Updating osd.3 2024-02-10T05:50:38.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:38 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:38.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:38 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:50:38.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:38 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:50:38.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:38 smithi192 ceph-mon[52881]: Deploying daemon osd.3 on smithi028 2024-02-10T05:50:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 243748 10.29 MB/sec execute 453 sec latency 71.799 ms 2024-02-10T05:50:39.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:39 smithi028 ceph-mon[66336]: pgmap v167: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.0 MiB/s wr, 425 op/s 2024-02-10T05:50:39.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:39 smithi192 ceph-mon[52881]: pgmap v167: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.0 MiB/s wr, 425 op/s 2024-02-10T05:50:40.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 244178 10.28 MB/sec execute 454 sec latency 41.128 ms 2024-02-10T05:50:41.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 244468 10.28 MB/sec execute 455 sec latency 45.288 ms 2024-02-10T05:50:41.441 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:41 smithi028 ceph-mon[66336]: pgmap v168: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.9 MiB/s wr, 417 op/s 2024-02-10T05:50:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:41 smithi192 ceph-mon[52881]: pgmap v168: 97 pgs: 97 active+clean; 337 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.9 MiB/s wr, 417 op/s 2024-02-10T05:50:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 244829 10.29 MB/sec execute 456 sec latency 38.673 ms 2024-02-10T05:50:42.977 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:42 smithi028 systemd[1]: Stopping Ceph osd.3 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:50:43.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 245154 10.30 MB/sec execute 457 sec latency 66.993 ms 2024-02-10T05:50:43.227 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:43 smithi028 ceph-mon[66336]: pgmap v169: 97 pgs: 97 active+clean; 327 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.1 MiB/s wr, 546 op/s 2024-02-10T05:50:43.228 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:43 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3[46179]: 2024-02-10T05:50:43.213+0000 7fe128d18700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:50:43.228 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:43 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3[46179]: 2024-02-10T05:50:43.213+0000 7fe128d18700 -1 osd.3 114 *** Got signal Terminated *** 2024-02-10T05:50:43.228 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:43 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3[46179]: 2024-02-10T05:50:43.213+0000 7fe128d18700 -1 osd.3 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:50:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:43 smithi192 ceph-mon[52881]: pgmap v169: 97 pgs: 97 active+clean; 327 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.1 MiB/s wr, 546 op/s 2024-02-10T05:50:44.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 245483 10.30 MB/sec execute 458 sec latency 60.644 ms 2024-02-10T05:50:44.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:44 smithi028 ceph-mon[66336]: osd.3 marked itself down and dead 2024-02-10T05:50:44.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:44 smithi192 ceph-mon[52881]: osd.3 marked itself down and dead 2024-02-10T05:50:45.147 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:44 smithi028 podman[79858]: 2024-02-10 05:50:44.865377582 +0000 UTC m=+1.792154668 container died 3ea0a9621c0ca7165f1ff5ff5881f5482ea78db34adbcc1e01abde6ff51c319d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3, io.buildah.version=1.19.8, release=754, io.openshift.expose-services=, distribution-scope=public, GIT_CLEAN=True, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.component=centos-stream-container, name=centos-stream, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, ceph=True, version=8) 2024-02-10T05:50:45.172 INFO:teuthology.orchestra.run.smithi028.stdout: 5 245495 10.28 MB/sec execute 459 sec latency 990.836 ms 2024-02-10T05:50:45.396 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:45 smithi028 podman[79858]: 2024-02-10 05:50:45.351995839 +0000 UTC m=+2.278772927 container cleanup 3ea0a9621c0ca7165f1ff5ff5881f5482ea78db34adbcc1e01abde6ff51c319d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, vcs-type=git, GIT_CLEAN=True, RELEASE=HEAD, name=centos-stream, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, release=754, ceph=True, version=8, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0) 2024-02-10T05:50:45.396 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:45 smithi028 bash[79858]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3 2024-02-10T05:50:45.396 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:45 smithi028 ceph-mon[66336]: pgmap v170: 97 pgs: 97 active+clean; 327 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.2 MiB/s wr, 399 op/s 2024-02-10T05:50:45.396 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:45 smithi028 ceph-mon[66336]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:50:45.396 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:45 smithi028 ceph-mon[66336]: failed to encode map e115 with expected crc 2024-02-10T05:50:45.396 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:45 smithi028 ceph-mon[66336]: osdmap e115: 8 total, 7 up, 8 in 2024-02-10T05:50:45.396 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:45 smithi028 ceph-mon[66336]: failed to encode map e115 with expected crc 2024-02-10T05:50:45.396 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:45 smithi028 ceph-mon[66336]: failed to encode map e115 with expected crc 2024-02-10T05:50:45.396 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:45 smithi028 ceph-mon[66336]: failed to encode map e115 with expected crc 2024-02-10T05:50:45.396 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:45.397 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:50:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:45 smithi192 ceph-mon[52881]: pgmap v170: 97 pgs: 97 active+clean; 327 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.2 MiB/s wr, 399 op/s 2024-02-10T05:50:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:45 smithi192 ceph-mon[52881]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:50:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:45 smithi192 ceph-mon[52881]: failed to encode map e115 with expected crc 2024-02-10T05:50:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:45 smithi192 ceph-mon[52881]: osdmap e115: 8 total, 7 up, 8 in 2024-02-10T05:50:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:45 smithi192 ceph-mon[52881]: failed to encode map e115 with expected crc 2024-02-10T05:50:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:45 smithi192 ceph-mon[52881]: failed to encode map e115 with expected crc 2024-02-10T05:50:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:45 smithi192 ceph-mon[52881]: failed to encode map e115 with expected crc 2024-02-10T05:50:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:50:45.672 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:45 smithi028 podman[79904]: 2024-02-10 05:50:45.62020836 +0000 UTC m=+0.748397388 container remove 3ea0a9621c0ca7165f1ff5ff5881f5482ea78db34adbcc1e01abde6ff51c319d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3, release=754, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, distribution-scope=public, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-10T05:50:46.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 245727 10.28 MB/sec execute 460 sec latency 1085.151 ms 2024-02-10T05:50:46.185 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:46 smithi028 podman[79990]: 2024-02-10 05:50:45.943127301 +0000 UTC m=+0.027456790 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:50:46.185 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:46 smithi028 podman[79990]: 2024-02-10 05:50:46.071344265 +0000 UTC m=+0.155673738 container create ec9204e8b66ae12e20a48a6d30ab94c9b66ff1c5d6d1034aa6725b411aa612e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.vendor=CentOS) 2024-02-10T05:50:46.422 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:46 smithi028 podman[79990]: 2024-02-10 05:50:46.324214639 +0000 UTC m=+0.408544106 container init ec9204e8b66ae12e20a48a6d30ab94c9b66ff1c5d6d1034aa6725b411aa612e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-deactivate, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-10T05:50:46.423 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:46 smithi028 podman[79990]: 2024-02-10 05:50:46.330382449 +0000 UTC m=+0.414711924 container start ec9204e8b66ae12e20a48a6d30ab94c9b66ff1c5d6d1034aa6725b411aa612e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-deactivate, RELEASE=main-290212a, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:50:46.423 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:46 smithi028 ceph-mon[66336]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.423 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:46 smithi028 ceph-mon[66336]: osdmap e116: 8 total, 7 up, 8 in 2024-02-10T05:50:46.423 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:46 smithi028 ceph-mon[66336]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.423 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:46 smithi028 ceph-mon[66336]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.423 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:46 smithi028 ceph-mon[66336]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.423 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:46 smithi028 ceph-mon[66336]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.423 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:46 smithi028 ceph-mon[66336]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.423 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:46 smithi028 ceph-mon[66336]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.424 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:46 smithi028 ceph-mon[66336]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.436 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:46 smithi028 podman[79990]: 2024-02-10 05:50:46.424254417 +0000 UTC m=+0.508583891 container attach ec9204e8b66ae12e20a48a6d30ab94c9b66ff1c5d6d1034aa6725b411aa612e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-10T05:50:46.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:46 smithi192 ceph-mon[52881]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:46 smithi192 ceph-mon[52881]: osdmap e116: 8 total, 7 up, 8 in 2024-02-10T05:50:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:46 smithi192 ceph-mon[52881]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:46 smithi192 ceph-mon[52881]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:46 smithi192 ceph-mon[52881]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:46 smithi192 ceph-mon[52881]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:46 smithi192 ceph-mon[52881]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:46 smithi192 ceph-mon[52881]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:46 smithi192 ceph-mon[52881]: failed to encode map e116 with expected crc 2024-02-10T05:50:46.947 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:46 smithi028 podman[79990]: 2024-02-10 05:50:46.732729539 +0000 UTC m=+0.817059015 container died ec9204e8b66ae12e20a48a6d30ab94c9b66ff1c5d6d1034aa6725b411aa612e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-10T05:50:47.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 245923 10.26 MB/sec execute 461 sec latency 45.583 ms 2024-02-10T05:50:47.463 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:47 smithi028 ceph-mon[66336]: pgmap v173: 97 pgs: 11 active+undersized+degraded, 15 peering, 2 stale+active+clean, 8 active+undersized, 61 active+clean; 336 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.6 MiB/s wr, 596 op/s; 94/1668 objects degraded (5.635%) 2024-02-10T05:50:47.463 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:47 smithi028 ceph-mon[66336]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:50:47.463 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:47 smithi028 ceph-mon[66336]: Health check failed: Degraded data redundancy: 94/1668 objects degraded (5.635%), 11 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:47 smithi192 ceph-mon[52881]: pgmap v173: 97 pgs: 11 active+undersized+degraded, 15 peering, 2 stale+active+clean, 8 active+undersized, 61 active+clean; 336 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.6 MiB/s wr, 596 op/s; 94/1668 objects degraded (5.635%) 2024-02-10T05:50:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:47 smithi192 ceph-mon[52881]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:50:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:47 smithi192 ceph-mon[52881]: Health check failed: Degraded data redundancy: 94/1668 objects degraded (5.635%), 11 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:47.719 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:47 smithi028 podman[80016]: 2024-02-10 05:50:47.690063858 +0000 UTC m=+0.943628393 container remove ec9204e8b66ae12e20a48a6d30ab94c9b66ff1c5d6d1034aa6725b411aa612e8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, RELEASE=main-290212a) 2024-02-10T05:50:47.971 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:47 smithi028 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.3.service: Deactivated successfully. 2024-02-10T05:50:47.971 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:47 smithi028 systemd[1]: Stopped Ceph osd.3 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:50:47.971 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:47 smithi028 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.3.service: Consumed 5min 1.352s CPU time. 2024-02-10T05:50:48.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 246146 10.24 MB/sec execute 462 sec latency 55.099 ms 2024-02-10T05:50:48.230 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:48 smithi028 systemd[1]: Starting Ceph osd.3 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:50:48.482 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:48 smithi028 podman[80117]: 2024-02-10 05:50:48.200208363 +0000 UTC m=+0.021483310 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:50:48.483 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:48 smithi028 podman[80117]: 2024-02-10 05:50:48.469281379 +0000 UTC m=+0.290556305 container create 3e89757cf485dd81f93e4204cc67c5cbbca0463366c03fbdb8fc143bda5c1a80 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, ceph=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-10T05:50:49.024 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:48 smithi028 ceph-mon[66336]: pgmap v174: 97 pgs: 11 active+undersized+degraded, 15 peering, 2 stale+active+clean, 8 active+undersized, 61 active+clean; 336 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.1 MiB/s wr, 528 op/s; 94/1668 objects degraded (5.635%) 2024-02-10T05:50:49.024 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:48 smithi028 podman[80117]: 2024-02-10 05:50:48.780369146 +0000 UTC m=+0.601644072 container init 3e89757cf485dd81f93e4204cc67c5cbbca0463366c03fbdb8fc143bda5c1a80 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-10T05:50:49.024 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:48 smithi028 podman[80117]: 2024-02-10 05:50:48.787376271 +0000 UTC m=+0.608651209 container start 3e89757cf485dd81f93e4204cc67c5cbbca0463366c03fbdb8fc143bda5c1a80 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-10T05:50:49.024 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:48 smithi028 podman[80117]: 2024-02-10 05:50:48.895545646 +0000 UTC m=+0.716820584 container attach 3e89757cf485dd81f93e4204cc67c5cbbca0463366c03fbdb8fc143bda5c1a80 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-10T05:50:49.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 246354 10.23 MB/sec execute 463 sec latency 61.041 ms 2024-02-10T05:50:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:48 smithi192 ceph-mon[52881]: pgmap v174: 97 pgs: 11 active+undersized+degraded, 15 peering, 2 stale+active+clean, 8 active+undersized, 61 active+clean; 336 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.1 MiB/s wr, 528 op/s; 94/1668 objects degraded (5.635%) 2024-02-10T05:50:49.535 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate[80132]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T05:50:49.535 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate[80132]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T05:50:49.535 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 bash[80117]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T05:50:49.535 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 bash[80117]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T05:50:49.535 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 bash[80117]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T05:50:49.535 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 bash[80117]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T05:50:49.536 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 bash[80117]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-10T05:50:49.536 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 bash[80117]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T05:50:49.536 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 bash[80117]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-10T05:50:49.536 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate[80132]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T05:50:49.536 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate[80132]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T05:50:49.536 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate[80132]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-10T05:50:49.536 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate[80132]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T05:50:49.536 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate[80132]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-10T05:50:49.536 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:49 smithi028 podman[80117]: 2024-02-10 05:50:49.32740625 +0000 UTC m=+1.148681168 container died 3e89757cf485dd81f93e4204cc67c5cbbca0463366c03fbdb8fc143bda5c1a80 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-10T05:50:50.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 246598 10.22 MB/sec execute 464 sec latency 81.807 ms 2024-02-10T05:50:50.295 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:50 smithi028 podman[80253]: 2024-02-10 05:50:50.284508674 +0000 UTC m=+0.952005222 container remove 3e89757cf485dd81f93e4204cc67c5cbbca0463366c03fbdb8fc143bda5c1a80 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-activate, RELEASE=main-290212a, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:50:50.548 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:50 smithi028 podman[80313]: 2024-02-10 05:50:50.428773754 +0000 UTC m=+0.021734970 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:50:50.548 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:50 smithi028 podman[80313]: 2024-02-10 05:50:50.527135128 +0000 UTC m=+0.120096331 container create 25f4cf94969eca9d6ac3da9d7362c35a8fe2151dfec3a497c81199f0994ef3f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-02-10T05:50:51.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 246896 10.22 MB/sec execute 465 sec latency 94.842 ms 2024-02-10T05:50:51.192 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:51 smithi028 ceph-mon[66336]: pgmap v175: 97 pgs: 16 active+undersized+degraded, 15 peering, 10 active+undersized, 56 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.1 MiB/s wr, 238 op/s; 131/1695 objects degraded (7.729%) 2024-02-10T05:50:51.192 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:51 smithi028 podman[80313]: 2024-02-10 05:50:51.031289794 +0000 UTC m=+0.624251026 container init 25f4cf94969eca9d6ac3da9d7362c35a8fe2151dfec3a497c81199f0994ef3f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.2) 2024-02-10T05:50:51.192 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:51 smithi028 podman[80313]: 2024-02-10 05:50:51.03486193 +0000 UTC m=+0.627823140 container start 25f4cf94969eca9d6ac3da9d7362c35a8fe2151dfec3a497c81199f0994ef3f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:50:51.193 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:51 smithi028 ceph-osd[80333]: -- 172.21.15.28:0/4265691953 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55be9db191e0 con 0x55be9daeac00 2024-02-10T05:50:51.193 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:51 smithi028 bash[80313]: 25f4cf94969eca9d6ac3da9d7362c35a8fe2151dfec3a497c81199f0994ef3f8 2024-02-10T05:50:51.193 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:51 smithi028 systemd[1]: Started Ceph osd.3 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:50:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:51 smithi192 ceph-mon[52881]: pgmap v175: 97 pgs: 16 active+undersized+degraded, 15 peering, 10 active+undersized, 56 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.1 MiB/s wr, 238 op/s; 131/1695 objects degraded (7.729%) 2024-02-10T05:50:51.750 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:51 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3[80329]: 2024-02-10T05:50:51.606+0000 7f8c678037c0 -1 Falling back to public interface 2024-02-10T05:50:52.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 247237 10.21 MB/sec execute 466 sec latency 94.151 ms 2024-02-10T05:50:52.518 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:52 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:52.518 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:52 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:52.519 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:52 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:52.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:52 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:52.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:52 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:50:52.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:52 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:53.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 247690 10.20 MB/sec execute 467 sec latency 59.926 ms 2024-02-10T05:50:53.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:53 smithi028 ceph-mon[66336]: pgmap v176: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 490 op/s; 207/1551 objects degraded (13.346%) 2024-02-10T05:50:53.282 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:53 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 207/1551 objects degraded (13.346%), 27 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:53.282 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:53 smithi028 ceph-mon[66336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T05:50:53.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:53 smithi192 ceph-mon[52881]: pgmap v176: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 490 op/s; 207/1551 objects degraded (13.346%) 2024-02-10T05:50:53.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:53 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 207/1551 objects degraded (13.346%), 27 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:53.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:53 smithi192 ceph-mon[52881]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T05:50:54.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 248011 10.19 MB/sec execute 468 sec latency 67.489 ms 2024-02-10T05:50:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 248394 10.21 MB/sec execute 469 sec latency 48.007 ms 2024-02-10T05:50:55.327 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:55 smithi028 ceph-mon[66336]: pgmap v177: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 414 op/s; 204/1533 objects degraded (13.307%) 2024-02-10T05:50:55.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:55 smithi192 ceph-mon[52881]: pgmap v177: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 329 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 414 op/s; 204/1533 objects degraded (13.307%) 2024-02-10T05:50:56.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 248786 10.22 MB/sec execute 470 sec latency 38.557 ms 2024-02-10T05:50:57.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 249130 10.23 MB/sec execute 471 sec latency 51.021 ms 2024-02-10T05:50:57.408 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:57 smithi028 ceph-mon[66336]: pgmap v178: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.0 MiB/s wr, 471 op/s; 212/1659 objects degraded (12.779%) 2024-02-10T05:50:57.409 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:57 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3[80329]: 2024-02-10T05:50:57.381+0000 7f8c678037c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-02-10T05:50:57.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:57 smithi192 ceph-mon[52881]: pgmap v178: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.0 MiB/s wr, 471 op/s; 212/1659 objects degraded (12.779%) 2024-02-10T05:50:58.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 249317 10.22 MB/sec execute 472 sec latency 55.242 ms 2024-02-10T05:50:58.439 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:58 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 212/1659 objects degraded (12.779%), 27 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:58.439 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:50:58.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:58 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 212/1659 objects degraded (12.779%), 27 pgs degraded (PG_DEGRADED) 2024-02-10T05:50:59.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 249471 10.20 MB/sec execute 473 sec latency 89.095 ms 2024-02-10T05:50:59.456 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:50:59 smithi028 ceph-mon[66336]: pgmap v179: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.7 MiB/s wr, 392 op/s; 212/1659 objects degraded (12.779%) 2024-02-10T05:50:59.457 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:50:59 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3[80329]: 2024-02-10T05:50:59.309+0000 7f8c678037c0 -1 osd.3 114 log_to_monitors true 2024-02-10T05:50:59.535 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:50:59.535 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (7m) 47s ago 16m 15.3M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:50:59.535 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (3m) 47s ago 16m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:50:59.535 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (3m) 3m ago 15m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:50:59.535 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (6m) 47s ago 16m 48.1M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:50:59.535 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (10m) 47s ago 10m 38.0M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:50:59.535 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (10m) 3m ago 10m 14.4M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:50:59.535 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (6m) 47s ago 17m 503M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:50:59.535 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (5m) 3m ago 15m 451M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (4m) 47s ago 17m 49.1M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (4m) 3m ago 15m 34.9M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (10m) 47s ago 10m 279M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (10m) 3m ago 10m 83.5M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (6m) 47s ago 16m 11.7M - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (6m) 3m ago 15m 21.1M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (2m) 47s ago 14m 313M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (105s) 47s ago 14m 253M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (57s) 47s ago 13m 11.6M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 a45f0e24ae44 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 starting - - - 1937M 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (13m) 3m ago 13m 567M 2704M 17.2.0 e1d6a67b021e 087422e9038c 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (12m) 3m ago 12m 430M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (12m) 3m ago 12m 331M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:50:59.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (11m) 3m ago 11m 523M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:50:59.537 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (5m) 47s ago 15m 61.9M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:50:59.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:50:59 smithi192 ceph-mon[52881]: pgmap v179: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.7 MiB/s wr, 392 op/s; 212/1659 objects degraded (12.779%) 2024-02-10T05:51:00.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 249695 10.19 MB/sec execute 474 sec latency 60.690 ms 2024-02-10T05:51:00.182 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:51:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:51:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:51:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:51:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:51:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:51:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:51:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:51:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-10T05:51:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 3 2024-02-10T05:51:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:51:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:51:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:51:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:51:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:51:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-10T05:51:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 7 2024-02-10T05:51:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:51:00.183 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:51:00.470 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:00 smithi028 ceph-mon[66336]: from='client.34326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:00.470 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:00 smithi028 ceph-mon[66336]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:00.470 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:00 smithi028 ceph-mon[66336]: from='osd.3 [v2:172.21.15.28:6826/948415501,v1:172.21.15.28:6827/948415501]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:51:00.470 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:00.470 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:51:00.470 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:00 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/981605410' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:00.470 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:51:00 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3[80329]: 2024-02-10T05:51:00.319+0000 7f8c58c5b700 -1 osd.3 114 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:51:00.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:00 smithi192 ceph-mon[52881]: from='client.34326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:00.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:00 smithi192 ceph-mon[52881]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:00.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:00 smithi192 ceph-mon[52881]: from='osd.3 [v2:172.21.15.28:6826/948415501,v1:172.21.15.28:6827/948415501]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:51:00.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:00.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:51:00.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:00 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/981605410' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:00.735 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:51:00.735 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:51:00.735 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:51:00.735 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:51:00.735 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:51:00.735 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:51:00.735 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:51:00.735 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:51:00.735 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:51:00.735 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "10/23 daemons upgraded", 2024-02-10T05:51:00.735 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T05:51:00.736 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:51:00.736 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:51:01.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 249898 10.17 MB/sec execute 475 sec latency 64.634 ms 2024-02-10T05:51:01.391 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_WARN Degraded data redundancy: 209/1641 objects degraded (12.736%), 27 pgs degraded 2024-02-10T05:51:01.391 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 209/1641 objects degraded (12.736%), 27 pgs degraded 2024-02-10T05:51:01.391 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-02-10T05:51:01.391 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.9 is active+undersized+degraded, acting [1,7] 2024-02-10T05:51:01.391 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.a is active+undersized+degraded, acting [1,2] 2024-02-10T05:51:01.391 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.d is active+undersized+degraded, acting [1,4] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.e is active+undersized+degraded, acting [2,7] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.12 is active+undersized+degraded, acting [5,7] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.14 is active+undersized+degraded, acting [6,5] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.19 is active+undersized+degraded, acting [6,0] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1b is active+undersized+degraded, acting [7,6] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.5 is active+undersized+degraded, acting [5,2] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.7 is active+undersized+degraded, acting [7,0] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.8 is active+undersized+degraded, acting [1,7] 2024-02-10T05:51:01.392 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.b is active+undersized+degraded, acting [0,4] 2024-02-10T05:51:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.c is active+undersized+degraded, acting [5,6] 2024-02-10T05:51:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2024-02-10T05:51:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.15 is active+undersized+degraded, acting [7,4] 2024-02-10T05:51:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2024-02-10T05:51:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.18 is active+undersized+degraded, acting [0,1] 2024-02-10T05:51:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.19 is active+undersized+degraded, acting [1,4] 2024-02-10T05:51:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1e is active+undersized+degraded, acting [6,2] 2024-02-10T05:51:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: pg 4.0 is active+undersized+degraded, acting [7,0] 2024-02-10T05:51:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: pg 4.14 is active+undersized+degraded, acting [2,7] 2024-02-10T05:51:01.393 INFO:teuthology.orchestra.run.smithi028.stdout: pg 4.19 is active+undersized+degraded, acting [2,0] 2024-02-10T05:51:01.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:01 smithi028 ceph-mon[66336]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:01.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:01 smithi028 ceph-mon[66336]: pgmap v180: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.8 MiB/s wr, 397 op/s; 209/1641 objects degraded (12.736%) 2024-02-10T05:51:01.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:01 smithi028 ceph-mon[66336]: from='osd.3 [v2:172.21.15.28:6826/948415501,v1:172.21.15.28:6827/948415501]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:51:01.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:01 smithi028 ceph-mon[66336]: failed to encode map e117 with expected crc 2024-02-10T05:51:01.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:01 smithi028 ceph-mon[66336]: osdmap e117: 8 total, 7 up, 8 in 2024-02-10T05:51:01.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:01 smithi028 ceph-mon[66336]: from='osd.3 [v2:172.21.15.28:6826/948415501,v1:172.21.15.28:6827/948415501]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:51:01.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:01 smithi028 ceph-mon[66336]: failed to encode map e117 with expected crc 2024-02-10T05:51:01.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:01 smithi028 ceph-mon[66336]: failed to encode map e117 with expected crc 2024-02-10T05:51:01.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:01.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:01.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:01 smithi192 ceph-mon[52881]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:01.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:01 smithi192 ceph-mon[52881]: pgmap v180: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.8 MiB/s wr, 397 op/s; 209/1641 objects degraded (12.736%) 2024-02-10T05:51:01.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:01 smithi192 ceph-mon[52881]: from='osd.3 [v2:172.21.15.28:6826/948415501,v1:172.21.15.28:6827/948415501]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:51:01.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:01 smithi192 ceph-mon[52881]: failed to encode map e117 with expected crc 2024-02-10T05:51:01.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:01 smithi192 ceph-mon[52881]: osdmap e117: 8 total, 7 up, 8 in 2024-02-10T05:51:01.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:01 smithi192 ceph-mon[52881]: from='osd.3 [v2:172.21.15.28:6826/948415501,v1:172.21.15.28:6827/948415501]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-10T05:51:01.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:01 smithi192 ceph-mon[52881]: failed to encode map e117 with expected crc 2024-02-10T05:51:01.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:01 smithi192 ceph-mon[52881]: failed to encode map e117 with expected crc 2024-02-10T05:51:01.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:01.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 249916 10.15 MB/sec execute 476 sec latency 904.369 ms 2024-02-10T05:51:02.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:02 smithi028 ceph-mon[66336]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:02.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:02 smithi028 ceph-mon[66336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:51:02.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:02 smithi028 ceph-mon[66336]: failed to encode map e118 with expected crc 2024-02-10T05:51:02.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:02 smithi028 ceph-mon[66336]: osd.3 [v2:172.21.15.28:6826/948415501,v1:172.21.15.28:6827/948415501] boot 2024-02-10T05:51:02.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:02 smithi028 ceph-mon[66336]: osdmap e118: 8 total, 8 up, 8 in 2024-02-10T05:51:02.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:02 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:51:02.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:02 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2763059702' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:51:02.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:02 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:02.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:02 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:02.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:02 smithi028 ceph-mon[66336]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T05:51:02.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:02 smithi192 ceph-mon[52881]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:02.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:02 smithi192 ceph-mon[52881]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:51:02.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:02 smithi192 ceph-mon[52881]: failed to encode map e118 with expected crc 2024-02-10T05:51:02.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:02 smithi192 ceph-mon[52881]: osd.3 [v2:172.21.15.28:6826/948415501,v1:172.21.15.28:6827/948415501] boot 2024-02-10T05:51:02.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:02 smithi192 ceph-mon[52881]: osdmap e118: 8 total, 8 up, 8 in 2024-02-10T05:51:02.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:02 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:51:02.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:02 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2763059702' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:51:02.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:02 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:02.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:02 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:02.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:02 smithi192 ceph-mon[52881]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T05:51:03.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 249916 10.13 MB/sec execute 477 sec latency 1904.416 ms 2024-02-10T05:51:03.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:03 smithi028 ceph-mon[66336]: pgmap v183: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.3 MiB/s wr, 500 op/s; 220/1710 objects degraded (12.865%) 2024-02-10T05:51:03.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:03 smithi028 ceph-mon[66336]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:03 smithi028 ceph-mon[66336]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:03 smithi028 ceph-mon[66336]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:03 smithi028 ceph-mon[66336]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:03 smithi028 ceph-mon[66336]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:03 smithi028 ceph-mon[66336]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:03 smithi028 ceph-mon[66336]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:03 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 209/1641 objects degraded (12.736%), 27 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:03.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:03 smithi028 ceph-mon[66336]: osdmap e120: 8 total, 8 up, 8 in 2024-02-10T05:51:03.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:03 smithi192 ceph-mon[52881]: pgmap v183: 97 pgs: 27 active+undersized+degraded, 14 active+undersized, 56 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 5.3 MiB/s wr, 500 op/s; 220/1710 objects degraded (12.865%) 2024-02-10T05:51:03.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:03 smithi192 ceph-mon[52881]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:03 smithi192 ceph-mon[52881]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:03 smithi192 ceph-mon[52881]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:03 smithi192 ceph-mon[52881]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:03 smithi192 ceph-mon[52881]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:03 smithi192 ceph-mon[52881]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:03 smithi192 ceph-mon[52881]: failed to encode map e119 with expected crc 2024-02-10T05:51:03.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:03 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 209/1641 objects degraded (12.736%), 27 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:03.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:03 smithi192 ceph-mon[52881]: osdmap e120: 8 total, 8 up, 8 in 2024-02-10T05:51:04.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 250015 10.11 MB/sec execute 478 sec latency 2337.433 ms 2024-02-10T05:51:04.377 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:04 smithi028 ceph-mon[66336]: failed to encode map e120 with expected crc 2024-02-10T05:51:04.378 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:04 smithi028 ceph-mon[66336]: failed to encode map e120 with expected crc 2024-02-10T05:51:04.378 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:04 smithi028 ceph-mon[66336]: failed to encode map e120 with expected crc 2024-02-10T05:51:04.378 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:04 smithi028 ceph-mon[66336]: failed to encode map e120 with expected crc 2024-02-10T05:51:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:04 smithi192 ceph-mon[52881]: failed to encode map e120 with expected crc 2024-02-10T05:51:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:04 smithi192 ceph-mon[52881]: failed to encode map e120 with expected crc 2024-02-10T05:51:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:04 smithi192 ceph-mon[52881]: failed to encode map e120 with expected crc 2024-02-10T05:51:04.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:04 smithi192 ceph-mon[52881]: failed to encode map e120 with expected crc 2024-02-10T05:51:05.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 250045 10.09 MB/sec execute 479 sec latency 890.820 ms 2024-02-10T05:51:05.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:05 smithi028 ceph-mon[66336]: pgmap v186: 97 pgs: 23 active+undersized+degraded, 13 active+undersized, 61 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.6 MiB/s wr, 393 op/s; 199/1692 objects degraded (11.761%); 30 KiB/s, 4 objects/s recovering 2024-02-10T05:51:05.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:05 smithi028 ceph-mon[66336]: failed to encode map e121 with expected crc 2024-02-10T05:51:05.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:05 smithi028 ceph-mon[66336]: osdmap e121: 8 total, 8 up, 8 in 2024-02-10T05:51:05.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:05 smithi028 ceph-mon[66336]: failed to encode map e121 with expected crc 2024-02-10T05:51:05.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:05 smithi028 ceph-mon[66336]: failed to encode map e121 with expected crc 2024-02-10T05:51:05.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:05 smithi028 ceph-mon[66336]: failed to encode map e121 with expected crc 2024-02-10T05:51:05.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:05 smithi192 ceph-mon[52881]: pgmap v186: 97 pgs: 23 active+undersized+degraded, 13 active+undersized, 61 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.6 MiB/s wr, 393 op/s; 199/1692 objects degraded (11.761%); 30 KiB/s, 4 objects/s recovering 2024-02-10T05:51:05.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:05 smithi192 ceph-mon[52881]: failed to encode map e121 with expected crc 2024-02-10T05:51:05.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:05 smithi192 ceph-mon[52881]: osdmap e121: 8 total, 8 up, 8 in 2024-02-10T05:51:05.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:05 smithi192 ceph-mon[52881]: failed to encode map e121 with expected crc 2024-02-10T05:51:05.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:05 smithi192 ceph-mon[52881]: failed to encode map e121 with expected crc 2024-02-10T05:51:05.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:05 smithi192 ceph-mon[52881]: failed to encode map e121 with expected crc 2024-02-10T05:51:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 250228 10.08 MB/sec execute 480 sec latency 1048.068 ms 2024-02-10T05:51:06.356 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:06 smithi028 ceph-mon[66336]: failed to encode map e121 with expected crc 2024-02-10T05:51:06.357 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:06 smithi028 ceph-mon[66336]: osdmap e122: 8 total, 8 up, 8 in 2024-02-10T05:51:06.357 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:06 smithi028 ceph-mon[66336]: failed to encode map e122 with expected crc 2024-02-10T05:51:06.357 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:06 smithi028 ceph-mon[66336]: failed to encode map e122 with expected crc 2024-02-10T05:51:06.357 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:06 smithi028 ceph-mon[66336]: failed to encode map e122 with expected crc 2024-02-10T05:51:06.357 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:06 smithi028 ceph-mon[66336]: failed to encode map e122 with expected crc 2024-02-10T05:51:06.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:06 smithi192 ceph-mon[52881]: failed to encode map e121 with expected crc 2024-02-10T05:51:06.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:06 smithi192 ceph-mon[52881]: osdmap e122: 8 total, 8 up, 8 in 2024-02-10T05:51:06.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:06 smithi192 ceph-mon[52881]: failed to encode map e122 with expected crc 2024-02-10T05:51:06.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:06 smithi192 ceph-mon[52881]: failed to encode map e122 with expected crc 2024-02-10T05:51:06.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:06 smithi192 ceph-mon[52881]: failed to encode map e122 with expected crc 2024-02-10T05:51:06.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:06 smithi192 ceph-mon[52881]: failed to encode map e122 with expected crc 2024-02-10T05:51:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 250536 10.08 MB/sec execute 481 sec latency 192.715 ms 2024-02-10T05:51:07.390 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:07 smithi028 ceph-mon[66336]: pgmap v189: 97 pgs: 3 peering, 1 active+undersized+degraded, 93 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 1.5 MiB/s wr, 206 op/s; 12/1359 objects degraded (0.883%); 3.7 MiB/s, 51 objects/s recovering 2024-02-10T05:51:07.390 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:07 smithi028 ceph-mon[66336]: failed to encode map e123 with expected crc 2024-02-10T05:51:07.390 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:07 smithi028 ceph-mon[66336]: failed to encode map e123 with expected crc 2024-02-10T05:51:07.390 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:07 smithi028 ceph-mon[66336]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T05:51:07.391 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:07 smithi028 ceph-mon[66336]: failed to encode map e123 with expected crc 2024-02-10T05:51:07.391 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:07 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:07 smithi192 ceph-mon[52881]: pgmap v189: 97 pgs: 3 peering, 1 active+undersized+degraded, 93 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 1.5 MiB/s wr, 206 op/s; 12/1359 objects degraded (0.883%); 3.7 MiB/s, 51 objects/s recovering 2024-02-10T05:51:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:07 smithi192 ceph-mon[52881]: failed to encode map e123 with expected crc 2024-02-10T05:51:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:07 smithi192 ceph-mon[52881]: failed to encode map e123 with expected crc 2024-02-10T05:51:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:07 smithi192 ceph-mon[52881]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T05:51:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:07 smithi192 ceph-mon[52881]: failed to encode map e123 with expected crc 2024-02-10T05:51:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:07 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:08.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 250848 10.07 MB/sec execute 482 sec latency 77.411 ms 2024-02-10T05:51:08.373 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: Upgrade: osd.4 is safe to restart 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 12/1359 objects degraded (0.883%), 1 pg degraded (PG_DEGRADED) 2024-02-10T05:51:08.374 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:08 smithi192 ceph-mon[52881]: pgmap v191: 97 pgs: 3 peering, 1 active+undersized+degraded, 93 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 1.2 MiB/s wr, 168 op/s; 12/1359 objects degraded (0.883%); 3.1 MiB/s, 40 objects/s recovering 2024-02-10T05:51:08.410 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:08.410 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:08.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:08.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:08.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:08.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:08.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:08.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:08.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:08.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T05:51:08.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-10T05:51:08.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: Upgrade: osd.4 is safe to restart 2024-02-10T05:51:08.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 12/1359 objects degraded (0.883%), 1 pg degraded (PG_DEGRADED) 2024-02-10T05:51:08.411 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:08 smithi028 ceph-mon[66336]: pgmap v191: 97 pgs: 3 peering, 1 active+undersized+degraded, 93 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 1.2 MiB/s wr, 168 op/s; 12/1359 objects degraded (0.883%); 3.1 MiB/s, 40 objects/s recovering 2024-02-10T05:51:09.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 251303 10.06 MB/sec execute 483 sec latency 54.404 ms 2024-02-10T05:51:10.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 251637 10.06 MB/sec execute 484 sec latency 35.123 ms 2024-02-10T05:51:10.449 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:10 smithi028 ceph-mon[66336]: Upgrade: Updating osd.4 2024-02-10T05:51:10.449 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:10.449 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:51:10.450 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:10.450 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:10 smithi028 ceph-mon[66336]: Deploying daemon osd.4 on smithi192 2024-02-10T05:51:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:10 smithi192 ceph-mon[52881]: Upgrade: Updating osd.4 2024-02-10T05:51:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:51:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:10 smithi192 ceph-mon[52881]: Deploying daemon osd.4 on smithi192 2024-02-10T05:51:11.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 252208 10.09 MB/sec execute 485 sec latency 36.130 ms 2024-02-10T05:51:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:11 smithi192 ceph-mon[52881]: pgmap v192: 97 pgs: 3 peering, 1 active+undersized+degraded, 93 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.1 MiB/s wr, 141 op/s; 12/1374 objects degraded (0.873%); 2.4 MiB/s, 31 objects/s recovering 2024-02-10T05:51:11.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:11 smithi028 ceph-mon[66336]: pgmap v192: 97 pgs: 3 peering, 1 active+undersized+degraded, 93 active+clean; 324 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.1 MiB/s wr, 141 op/s; 12/1374 objects degraded (0.873%); 2.4 MiB/s, 31 objects/s recovering 2024-02-10T05:51:12.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 252614 10.11 MB/sec execute 486 sec latency 47.141 ms 2024-02-10T05:51:12.520 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:12 smithi028 ceph-mon[66336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/1374 objects degraded (0.873%), 1 pg degraded) 2024-02-10T05:51:12.520 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:12 smithi028 ceph-mon[66336]: Cluster is now healthy 2024-02-10T05:51:12.557 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:12 smithi192 ceph-mon[52881]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/1374 objects degraded (0.873%), 1 pg degraded) 2024-02-10T05:51:12.557 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:12 smithi192 ceph-mon[52881]: Cluster is now healthy 2024-02-10T05:51:13.173 INFO:teuthology.orchestra.run.smithi028.stdout: 5 252864 10.10 MB/sec execute 487 sec latency 54.825 ms 2024-02-10T05:51:13.235 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:12 smithi192 systemd[1]: Stopping Ceph osd.4 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:51:13.533 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:13 smithi028 ceph-mon[66336]: pgmap v193: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.4 MiB/s wr, 437 op/s; 1.9 MiB/s, 26 objects/s recovering 2024-02-10T05:51:13.533 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:13 smithi028 ceph-mon[66336]: osd.4 marked itself down and dead 2024-02-10T05:51:13.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:13 smithi192 ceph-mon[52881]: pgmap v193: 97 pgs: 97 active+clean; 330 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.4 MiB/s wr, 437 op/s; 1.9 MiB/s, 26 objects/s recovering 2024-02-10T05:51:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:13 smithi192 ceph-mon[52881]: osd.4 marked itself down and dead 2024-02-10T05:51:13.572 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:13 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[31064]: 2024-02-10T05:51:13.254+0000 7f3127a51700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:51:13.572 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:13 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[31064]: 2024-02-10T05:51:13.254+0000 7f3127a51700 -1 osd.4 123 *** Got signal Terminated *** 2024-02-10T05:51:13.572 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:13 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[31064]: 2024-02-10T05:51:13.254+0000 7f3127a51700 -1 osd.4 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:51:14.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 253071 10.09 MB/sec execute 488 sec latency 302.020 ms 2024-02-10T05:51:14.550 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:14 smithi028 ceph-mon[66336]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:51:14.550 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:14 smithi028 ceph-mon[66336]: osdmap e124: 8 total, 7 up, 8 in 2024-02-10T05:51:14.550 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:14 smithi028 ceph-mon[66336]: failed to encode map e124 with expected crc 2024-02-10T05:51:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:14 smithi192 ceph-mon[52881]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:51:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:14 smithi192 ceph-mon[52881]: osdmap e124: 8 total, 7 up, 8 in 2024-02-10T05:51:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:14 smithi192 ceph-mon[52881]: failed to encode map e124 with expected crc 2024-02-10T05:51:14.572 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:14 smithi192 podman[57382]: 2024-02-10 05:51:14.477453711 +0000 UTC m=+1.339109440 container died 087422e9038c3a5c7a386490a3048f0edcba0df7768ea5eb22d075b0a35eb2cb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, release=754, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64) 2024-02-10T05:51:15.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:14 smithi192 podman[57382]: 2024-02-10 05:51:14.879057911 +0000 UTC m=+1.740713643 container cleanup 087422e9038c3a5c7a386490a3048f0edcba0df7768ea5eb22d075b0a35eb2cb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, com.redhat.component=centos-stream-container, GIT_CLEAN=True, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, distribution-scope=public, RELEASE=HEAD) 2024-02-10T05:51:15.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:14 smithi192 bash[57382]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4 2024-02-10T05:51:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 253257 10.07 MB/sec execute 489 sec latency 1022.034 ms 2024-02-10T05:51:15.399 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:15 smithi028 ceph-mon[66336]: failed to encode map e124 with expected crc 2024-02-10T05:51:15.400 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:15 smithi028 ceph-mon[66336]: failed to encode map e124 with expected crc 2024-02-10T05:51:15.400 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:15 smithi028 ceph-mon[66336]: pgmap v195: 97 pgs: 14 stale+active+clean, 83 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.4 KiB/s rd, 3.4 MiB/s wr, 395 op/s; 500 B/s, 1 objects/s recovering 2024-02-10T05:51:15.400 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:15 smithi028 ceph-mon[66336]: failed to encode map e125 with expected crc 2024-02-10T05:51:15.400 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:15 smithi028 ceph-mon[66336]: osdmap e125: 8 total, 7 up, 8 in 2024-02-10T05:51:15.400 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:15 smithi028 ceph-mon[66336]: failed to encode map e125 with expected crc 2024-02-10T05:51:15.400 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:15.400 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:51:15.543 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:15 smithi192 podman[57409]: 2024-02-10 05:51:15.189270996 +0000 UTC m=+0.710015477 container remove 087422e9038c3a5c7a386490a3048f0edcba0df7768ea5eb22d075b0a35eb2cb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, version=8, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_CLEAN=True, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-10T05:51:15.543 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:15 smithi192 ceph-mon[52881]: failed to encode map e124 with expected crc 2024-02-10T05:51:15.543 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:15 smithi192 ceph-mon[52881]: failed to encode map e124 with expected crc 2024-02-10T05:51:15.544 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:15 smithi192 ceph-mon[52881]: pgmap v195: 97 pgs: 14 stale+active+clean, 83 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.4 KiB/s rd, 3.4 MiB/s wr, 395 op/s; 500 B/s, 1 objects/s recovering 2024-02-10T05:51:15.544 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:15 smithi192 ceph-mon[52881]: failed to encode map e125 with expected crc 2024-02-10T05:51:15.544 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:15 smithi192 ceph-mon[52881]: osdmap e125: 8 total, 7 up, 8 in 2024-02-10T05:51:15.544 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:15 smithi192 ceph-mon[52881]: failed to encode map e125 with expected crc 2024-02-10T05:51:15.544 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:15.544 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:51:15.822 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:15 smithi192 podman[57501]: 2024-02-10 05:51:15.4477283 +0000 UTC m=+0.023131273 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:51:15.822 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:15 smithi192 podman[57501]: 2024-02-10 05:51:15.576675556 +0000 UTC m=+0.152078527 container create 63a7a9f26216c45057b1ceccb120fc068a0163794af60cecdcd717a74fa2faa8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-10T05:51:16.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 253479 10.05 MB/sec execute 490 sec latency 63.398 ms 2024-02-10T05:51:16.180 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:15 smithi192 podman[57501]: 2024-02-10 05:51:15.860648706 +0000 UTC m=+0.436051691 container init 63a7a9f26216c45057b1ceccb120fc068a0163794af60cecdcd717a74fa2faa8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T05:51:16.180 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:15 smithi192 podman[57501]: 2024-02-10 05:51:15.864077644 +0000 UTC m=+0.439480626 container start 63a7a9f26216c45057b1ceccb120fc068a0163794af60cecdcd717a74fa2faa8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD) 2024-02-10T05:51:16.180 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:15 smithi192 podman[57501]: 2024-02-10 05:51:15.960219238 +0000 UTC m=+0.535622225 container attach 63a7a9f26216c45057b1ceccb120fc068a0163794af60cecdcd717a74fa2faa8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-02-10T05:51:16.423 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:16 smithi028 ceph-mon[66336]: failed to encode map e125 with expected crc 2024-02-10T05:51:16.424 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:16 smithi028 ceph-mon[66336]: failed to encode map e125 with expected crc 2024-02-10T05:51:16.424 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:16 smithi028 ceph-mon[66336]: failed to encode map e125 with expected crc 2024-02-10T05:51:16.424 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:16 smithi028 ceph-mon[66336]: pgmap v197: 97 pgs: 24 active+undersized+degraded, 2 stale+active+clean, 10 active+undersized, 61 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.5 MiB/s wr, 515 op/s; 235/1689 objects degraded (13.914%); 500 B/s, 1 objects/s recovering 2024-02-10T05:51:16.572 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:16 smithi192 podman[57501]: 2024-02-10 05:51:16.18107261 +0000 UTC m=+0.756475585 container died 63a7a9f26216c45057b1ceccb120fc068a0163794af60cecdcd717a74fa2faa8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:51:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:16 smithi192 ceph-mon[52881]: failed to encode map e125 with expected crc 2024-02-10T05:51:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:16 smithi192 ceph-mon[52881]: failed to encode map e125 with expected crc 2024-02-10T05:51:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:16 smithi192 ceph-mon[52881]: failed to encode map e125 with expected crc 2024-02-10T05:51:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:16 smithi192 ceph-mon[52881]: pgmap v197: 97 pgs: 24 active+undersized+degraded, 2 stale+active+clean, 10 active+undersized, 61 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.5 MiB/s wr, 515 op/s; 235/1689 objects degraded (13.914%); 500 B/s, 1 objects/s recovering 2024-02-10T05:51:17.067 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:16 smithi192 podman[57527]: 2024-02-10 05:51:16.831738699 +0000 UTC m=+0.642828290 container remove 63a7a9f26216c45057b1ceccb120fc068a0163794af60cecdcd717a74fa2faa8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-10T05:51:17.067 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:16 smithi192 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-10T05:51:17.067 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:16 smithi192 systemd[1]: Stopped Ceph osd.4 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:51:17.067 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:16 smithi192 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.4.service: Consumed 4min 58.302s CPU time. 2024-02-10T05:51:17.067 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:17 smithi192 systemd[1]: Starting Ceph osd.4 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:51:17.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 253690 10.04 MB/sec execute 491 sec latency 49.077 ms 2024-02-10T05:51:17.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:17 smithi192 ceph-mon[52881]: Health check failed: Degraded data redundancy: 235/1689 objects degraded (13.914%), 24 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:17.322 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:17 smithi192 podman[57628]: 2024-02-10 05:51:17.199230497 +0000 UTC m=+0.017856070 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:51:17.481 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:17 smithi028 ceph-mon[66336]: Health check failed: Degraded data redundancy: 235/1689 objects degraded (13.914%), 24 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:17.729 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:17 smithi192 podman[57628]: 2024-02-10 05:51:17.350150658 +0000 UTC m=+0.168776266 container create 3aa218c51ea4efd203217120d2248111700855c15fc94626a3d695d905cc76fb (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-10T05:51:18.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:17 smithi192 podman[57628]: 2024-02-10 05:51:17.729096382 +0000 UTC m=+0.547721950 container init 3aa218c51ea4efd203217120d2248111700855c15fc94626a3d695d905cc76fb (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-290212a) 2024-02-10T05:51:18.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:17 smithi192 podman[57628]: 2024-02-10 05:51:17.733463936 +0000 UTC m=+0.552089506 container start 3aa218c51ea4efd203217120d2248111700855c15fc94626a3d695d905cc76fb (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux ) 2024-02-10T05:51:18.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:17 smithi192 podman[57628]: 2024-02-10 05:51:17.836361984 +0000 UTC m=+0.654987559 container attach 3aa218c51ea4efd203217120d2248111700855c15fc94626a3d695d905cc76fb (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate, RELEASE=main-290212a, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-10T05:51:18.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 253987 10.05 MB/sec execute 492 sec latency 62.976 ms 2024-02-10T05:51:18.572 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate[57643]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T05:51:18.572 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate[57643]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T05:51:18.572 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 bash[57628]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T05:51:18.572 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 bash[57628]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-10T05:51:18.573 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 bash[57628]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T05:51:18.573 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 bash[57628]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T05:51:18.573 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 bash[57628]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-10T05:51:18.573 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 bash[57628]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T05:51:18.573 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 bash[57628]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-10T05:51:18.573 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate[57643]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-10T05:51:18.573 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate[57643]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-10T05:51:18.573 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate[57643]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-10T05:51:18.573 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate[57643]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-10T05:51:18.573 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate[57643]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-10T05:51:18.573 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 podman[57759]: 2024-02-10 05:51:18.28738726 +0000 UTC m=+0.020111426 container died 3aa218c51ea4efd203217120d2248111700855c15fc94626a3d695d905cc76fb (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-10T05:51:18.891 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:18 smithi192 ceph-mon[52881]: pgmap v198: 97 pgs: 24 active+undersized+degraded, 2 stale+active+clean, 10 active+undersized, 61 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.4 MiB/s wr, 508 op/s; 235/1689 objects degraded (13.914%); 500 B/s, 1 objects/s recovering 2024-02-10T05:51:19.014 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:18 smithi028 ceph-mon[66336]: pgmap v198: 97 pgs: 24 active+undersized+degraded, 2 stale+active+clean, 10 active+undersized, 61 active+clean; 333 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.4 MiB/s wr, 508 op/s; 235/1689 objects degraded (13.914%); 500 B/s, 1 objects/s recovering 2024-02-10T05:51:19.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 254277 10.03 MB/sec execute 493 sec latency 113.627 ms 2024-02-10T05:51:19.275 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:18 smithi192 podman[57759]: 2024-02-10 05:51:18.891226436 +0000 UTC m=+0.623950608 container remove 3aa218c51ea4efd203217120d2248111700855c15fc94626a3d695d905cc76fb (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-10T05:51:19.275 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:19 smithi192 podman[57822]: 2024-02-10 05:51:19.024789017 +0000 UTC m=+0.020450195 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:51:19.275 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:19 smithi192 podman[57822]: 2024-02-10 05:51:19.275804732 +0000 UTC m=+0.271465949 container create b15b2733ec773aed85f8a85e156e9d722f991cbafb81a9740ca2e0293a9e4aad (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_BRANCH=HEAD) 2024-02-10T05:51:19.822 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:19 smithi192 podman[57822]: 2024-02-10 05:51:19.528591875 +0000 UTC m=+0.524253066 container init b15b2733ec773aed85f8a85e156e9d722f991cbafb81a9740ca2e0293a9e4aad (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:51:19.822 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:19 smithi192 podman[57822]: 2024-02-10 05:51:19.531862559 +0000 UTC m=+0.527523735 container start b15b2733ec773aed85f8a85e156e9d722f991cbafb81a9740ca2e0293a9e4aad (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:51:19.822 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:19 smithi192 ceph-osd[57840]: -- 172.21.15.192:0/3425387517 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561a109311e0 con 0x561a10902c00 2024-02-10T05:51:19.822 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:19 smithi192 bash[57822]: b15b2733ec773aed85f8a85e156e9d722f991cbafb81a9740ca2e0293a9e4aad 2024-02-10T05:51:19.822 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:19 smithi192 systemd[1]: Started Ceph osd.4 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:51:20.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 254722 10.03 MB/sec execute 494 sec latency 74.513 ms 2024-02-10T05:51:20.572 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:20 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:51:20.073+0000 7f1cf5bfe7c0 -1 Falling back to public interface 2024-02-10T05:51:20.849 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:20.849 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:20.850 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:20.850 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:20 smithi028 ceph-mon[66336]: pgmap v199: 97 pgs: 28 active+undersized+degraded, 13 active+undersized, 56 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 3.5 MiB/s wr, 255 op/s; 266/1644 objects degraded (16.180%) 2024-02-10T05:51:20.850 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:20 smithi192 ceph-mon[52881]: pgmap v199: 97 pgs: 28 active+undersized+degraded, 13 active+undersized, 56 active+clean; 332 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 3.5 MiB/s wr, 255 op/s; 266/1644 objects degraded (16.180%) 2024-02-10T05:51:21.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:21.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 254955 10.01 MB/sec execute 495 sec latency 44.408 ms 2024-02-10T05:51:22.137 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:22 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 249/1551 objects degraded (16.054%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:22.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 255297 10.02 MB/sec execute 496 sec latency 45.923 ms 2024-02-10T05:51:22.527 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:22 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 249/1551 objects degraded (16.054%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:23.151 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:23 smithi028 ceph-mon[66336]: pgmap v200: 97 pgs: 28 active+undersized+degraded, 13 active+undersized, 56 active+clean; 324 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.4 MiB/s wr, 466 op/s; 249/1551 objects degraded (16.054%) 2024-02-10T05:51:23.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 255588 10.02 MB/sec execute 497 sec latency 61.447 ms 2024-02-10T05:51:23.406 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:23 smithi192 ceph-mon[52881]: pgmap v200: 97 pgs: 28 active+undersized+degraded, 13 active+undersized, 56 active+clean; 324 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 4.4 MiB/s wr, 466 op/s; 249/1551 objects degraded (16.054%) 2024-02-10T05:51:24.175 INFO:teuthology.orchestra.run.smithi028.stdout: 5 256058 10.05 MB/sec execute 498 sec latency 40.196 ms 2024-02-10T05:51:25.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 256423 10.05 MB/sec execute 499 sec latency 30.329 ms 2024-02-10T05:51:25.189 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:24 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:25.190 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:24 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:25.190 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:24 smithi028 ceph-mon[66336]: pgmap v201: 97 pgs: 28 active+undersized+degraded, 13 active+undersized, 56 active+clean; 325 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.8 MiB/s wr, 390 op/s; 252/1566 objects degraded (16.092%) 2024-02-10T05:51:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:24 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:24 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:24 smithi192 ceph-mon[52881]: pgmap v201: 97 pgs: 28 active+undersized+degraded, 13 active+undersized, 56 active+clean; 325 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.8 MiB/s wr, 390 op/s; 252/1566 objects degraded (16.092%) 2024-02-10T05:51:26.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 256817 10.04 MB/sec execute 500 sec latency 40.975 ms 2024-02-10T05:51:26.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:26 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:26.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:26 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:26.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:26 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:26.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:26 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:27.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:26 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:51:26.959+0000 7f1cf5bfe7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-02-10T05:51:27.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 257214 10.03 MB/sec execute 501 sec latency 45.608 ms 2024-02-10T05:51:27.206 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:27 smithi028 ceph-mon[66336]: pgmap v202: 97 pgs: 28 active+undersized+degraded, 13 active+undersized, 56 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 538 op/s; 280/1704 objects degraded (16.432%) 2024-02-10T05:51:27.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:27 smithi192 ceph-mon[52881]: pgmap v202: 97 pgs: 28 active+undersized+degraded, 13 active+undersized, 56 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.8 MiB/s wr, 538 op/s; 280/1704 objects degraded (16.432%) 2024-02-10T05:51:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 257638 10.04 MB/sec execute 502 sec latency 47.239 ms 2024-02-10T05:51:28.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:28 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 280/1704 objects degraded (16.432%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:28.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:28 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 280/1704 objects degraded (16.432%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:28.822 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:28 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:51:28.541+0000 7f1cf5bfe7c0 -1 osd.4 123 log_to_monitors true 2024-02-10T05:51:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 258046 10.03 MB/sec execute 503 sec latency 75.831 ms 2024-02-10T05:51:29.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:29 smithi028 ceph-mon[66336]: pgmap v203: 97 pgs: 28 active+undersized+degraded, 13 active+undersized, 56 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.0 MiB/s wr, 419 op/s; 280/1704 objects degraded (16.432%) 2024-02-10T05:51:29.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:29 smithi028 ceph-mon[66336]: from='osd.4 [v2:172.21.15.192:6800/4080043109,v1:172.21.15.192:6801/4080043109]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:51:29.195 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:29 smithi028 ceph-mon[66336]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:51:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:29 smithi192 ceph-mon[52881]: pgmap v203: 97 pgs: 28 active+undersized+degraded, 13 active+undersized, 56 active+clean; 336 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.0 MiB/s wr, 419 op/s; 280/1704 objects degraded (16.432%) 2024-02-10T05:51:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:29 smithi192 ceph-mon[52881]: from='osd.4 [v2:172.21.15.192:6800/4080043109,v1:172.21.15.192:6801/4080043109]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:51:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:29 smithi192 ceph-mon[52881]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:51:30.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:51:29 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:51:29.681+0000 7f1ce304e700 -1 osd.4 123 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:51:30.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 258297 10.02 MB/sec execute 504 sec latency 165.779 ms 2024-02-10T05:51:30.420 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:51:30.420 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: failed to encode map e126 with expected crc 2024-02-10T05:51:30.420 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: osdmap e126: 8 total, 7 up, 8 in 2024-02-10T05:51:30.420 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: failed to encode map e126 with expected crc 2024-02-10T05:51:30.420 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='osd.4 [v2:172.21.15.192:6800/4080043109,v1:172.21.15.192:6801/4080043109]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:51:30.420 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:51:30.420 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:30.420 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:30.420 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:30.420 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:30.421 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:30.421 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:30.421 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='osd.4 ' entity='osd.4' 2024-02-10T05:51:30.421 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:30.421 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:30.421 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:30.421 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:30.421 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T05:51:30.421 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:30.421 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:51:30.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:51:30.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: failed to encode map e126 with expected crc 2024-02-10T05:51:30.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: osdmap e126: 8 total, 7 up, 8 in 2024-02-10T05:51:30.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: failed to encode map e126 with expected crc 2024-02-10T05:51:30.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='osd.4 [v2:172.21.15.192:6800/4080043109,v1:172.21.15.192:6801/4080043109]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:51:30.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:51:30.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:30.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:30.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:30.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:30.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:30.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:30.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='osd.4 ' entity='osd.4' 2024-02-10T05:51:30.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:30.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:30.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:30.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:30.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T05:51:30.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:30.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:51:31.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 258477 10.01 MB/sec execute 505 sec latency 483.221 ms 2024-02-10T05:51:31.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:31 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T05:51:31.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:31 smithi028 ceph-mon[66336]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-02-10T05:51:31.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:31 smithi028 ceph-mon[66336]: pgmap v205: 97 pgs: 28 active+undersized+degraded, 13 active+undersized, 56 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.9 MiB/s wr, 469 op/s; 288/1734 objects degraded (16.609%) 2024-02-10T05:51:31.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:31 smithi028 ceph-mon[66336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:51:31.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:31 smithi028 ceph-mon[66336]: osd.4 [v2:172.21.15.192:6800/4080043109,v1:172.21.15.192:6801/4080043109] boot 2024-02-10T05:51:31.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:31 smithi028 ceph-mon[66336]: osdmap e127: 8 total, 8 up, 8 in 2024-02-10T05:51:31.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:51:31.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:31 smithi028 ceph-mon[66336]: failed to encode map e127 with expected crc 2024-02-10T05:51:31.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:31 smithi028 ceph-mon[66336]: failed to encode map e127 with expected crc 2024-02-10T05:51:31.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:31 smithi028 ceph-mon[66336]: failed to encode map e127 with expected crc 2024-02-10T05:51:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:31 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T05:51:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:31 smithi192 ceph-mon[52881]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-02-10T05:51:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:31 smithi192 ceph-mon[52881]: pgmap v205: 97 pgs: 28 active+undersized+degraded, 13 active+undersized, 56 active+clean; 338 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.9 MiB/s wr, 469 op/s; 288/1734 objects degraded (16.609%) 2024-02-10T05:51:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:31 smithi192 ceph-mon[52881]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:51:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:31 smithi192 ceph-mon[52881]: osd.4 [v2:172.21.15.192:6800/4080043109,v1:172.21.15.192:6801/4080043109] boot 2024-02-10T05:51:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:31 smithi192 ceph-mon[52881]: osdmap e127: 8 total, 8 up, 8 in 2024-02-10T05:51:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:51:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:31 smithi192 ceph-mon[52881]: failed to encode map e127 with expected crc 2024-02-10T05:51:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:31 smithi192 ceph-mon[52881]: failed to encode map e127 with expected crc 2024-02-10T05:51:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:31 smithi192 ceph-mon[52881]: failed to encode map e127 with expected crc 2024-02-10T05:51:31.763 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:51:32.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 258501 9.99 MB/sec execute 506 sec latency 1447.843 ms 2024-02-10T05:51:32.492 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:32 smithi028 ceph-mon[66336]: failed to encode map e127 with expected crc 2024-02-10T05:51:32.492 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:32 smithi028 ceph-mon[66336]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.493 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:32 smithi028 ceph-mon[66336]: osdmap e128: 8 total, 8 up, 8 in 2024-02-10T05:51:32.493 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:32 smithi028 ceph-mon[66336]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.493 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:32 smithi028 ceph-mon[66336]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.493 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:32 smithi028 ceph-mon[66336]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.493 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:32 smithi028 ceph-mon[66336]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.493 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:32 smithi028 ceph-mon[66336]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.493 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:32 smithi028 ceph-mon[66336]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.493 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:51:32.493 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (7m) 31s ago 16m 15.4M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:51:32.493 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (3m) 31s ago 16m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (3m) 8s ago 16m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (7m) 31s ago 16m 45.8M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (10m) 31s ago 10m 38.5M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (10m) 8s ago 10m 15.0M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (6m) 31s ago 18m 504M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (5m) 8s ago 16m 453M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (5m) 31s ago 18m 51.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (4m) 8s ago 16m 41.1M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (10m) 31s ago 10m 282M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (10m) 8s ago 10m 86.2M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (7m) 31s ago 16m 13.5M - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (6m) 8s ago 15m 20.8M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:51:32.494 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (3m) 31s ago 15m 339M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:51:32.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (2m) 31s ago 14m 289M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:51:32.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (90s) 31s ago 14m 233M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 a45f0e24ae44 2024-02-10T05:51:32.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (42s) 31s ago 14m 11.6M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 25f4cf94969e 2024-02-10T05:51:32.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (13s) 8s ago 13m 11.5M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 b15b2733ec77 2024-02-10T05:51:32.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (13m) 8s ago 13m 599M 2704M 17.2.0 e1d6a67b021e ae5279672ff8 2024-02-10T05:51:32.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (12m) 8s ago 12m 513M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:51:32.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (12m) 8s ago 12m 812M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:51:32.495 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (5m) 31s ago 15m 62.1M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:51:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:32 smithi192 ceph-mon[52881]: failed to encode map e127 with expected crc 2024-02-10T05:51:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:32 smithi192 ceph-mon[52881]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:32 smithi192 ceph-mon[52881]: osdmap e128: 8 total, 8 up, 8 in 2024-02-10T05:51:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:32 smithi192 ceph-mon[52881]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:32 smithi192 ceph-mon[52881]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:32 smithi192 ceph-mon[52881]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:32 smithi192 ceph-mon[52881]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:32 smithi192 ceph-mon[52881]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:32 smithi192 ceph-mon[52881]: failed to encode map e128 with expected crc 2024-02-10T05:51:32.979 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:51:32.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:51:32.979 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:51:32.980 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-10T05:51:32.981 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 9 2024-02-10T05:51:32.981 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:51:32.981 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:51:33.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 258607 9.98 MB/sec execute 507 sec latency 2046.024 ms 2024-02-10T05:51:33.288 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: failed to encode map e128 with expected crc 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: failed to encode map e128 with expected crc 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: from='client.34354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: pgmap v208: 97 pgs: 14 peering, 14 active+undersized+degraded, 13 active+undersized, 56 active+clean; 332 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.0 MiB/s wr, 509 op/s; 144/1599 objects degraded (9.006%) 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: from='client.44295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.289 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 144/1599 objects degraded (9.006%), 14 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:33.290 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:33 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/76801760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:33.431 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:51:33.431 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:51:33.431 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:51:33.431 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:51:33.431 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:51:33.431 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:51:33.431 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:51:33.431 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:51:33.431 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:51:33.432 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "11/23 daemons upgraded", 2024-02-10T05:51:33.432 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T05:51:33.432 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:51:33.432 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:51:33.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: failed to encode map e128 with expected crc 2024-02-10T05:51:33.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: failed to encode map e128 with expected crc 2024-02-10T05:51:33.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: from='client.34354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:33.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: pgmap v208: 97 pgs: 14 peering, 14 active+undersized+degraded, 13 active+undersized, 56 active+clean; 332 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.0 MiB/s wr, 509 op/s; 144/1599 objects degraded (9.006%) 2024-02-10T05:51:33.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: from='client.44295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:33.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T05:51:33.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: failed to encode map e129 with expected crc 2024-02-10T05:51:33.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 144/1599 objects degraded (9.006%), 14 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:33.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:33 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/76801760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_WARN Degraded data redundancy: 144/1599 objects degraded (9.006%), 14 pgs degraded 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 144/1599 objects degraded (9.006%), 14 pgs degraded 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.6 is active+undersized+degraded, acting [1,6] 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.f is active+undersized+degraded, acting [0,7] 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.13 is active+undersized+degraded, acting [0,3] 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1f is active+undersized+degraded, acting [0,7] 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-02-10T05:51:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-02-10T05:51:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-02-10T05:51:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-02-10T05:51:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-02-10T05:51:34.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 258904 9.98 MB/sec execute 508 sec latency 85.127 ms 2024-02-10T05:51:34.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:34 smithi028 ceph-mon[66336]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:34.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:34 smithi028 ceph-mon[66336]: failed to encode map e129 with expected crc 2024-02-10T05:51:34.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:34 smithi028 ceph-mon[66336]: failed to encode map e129 with expected crc 2024-02-10T05:51:34.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:34 smithi028 ceph-mon[66336]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T05:51:34.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:34 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2482094767' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:51:34.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:34 smithi192 ceph-mon[52881]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:34.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:34 smithi192 ceph-mon[52881]: failed to encode map e129 with expected crc 2024-02-10T05:51:34.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:34 smithi192 ceph-mon[52881]: failed to encode map e129 with expected crc 2024-02-10T05:51:34.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:34 smithi192 ceph-mon[52881]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T05:51:34.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:34 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2482094767' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:51:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 259197 9.98 MB/sec execute 509 sec latency 65.443 ms 2024-02-10T05:51:35.378 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:35 smithi028 ceph-mon[66336]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:35.378 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:35 smithi028 ceph-mon[66336]: failed to encode map e130 with expected crc 2024-02-10T05:51:35.379 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:35 smithi028 ceph-mon[66336]: pgmap v211: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 14 peering, 11 active+undersized+degraded, 11 active+undersized, 60 active+clean; 326 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 5.1 MiB/s wr, 740 op/s; 142/1569 objects degraded (9.050%); 0 B/s, 72 keys/s, 0 objects/s recovering 2024-02-10T05:51:35.379 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:35 smithi028 ceph-mon[66336]: osdmap e131: 8 total, 8 up, 8 in 2024-02-10T05:51:35.379 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:35 smithi028 ceph-mon[66336]: failed to encode map e131 with expected crc 2024-02-10T05:51:35.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:35 smithi192 ceph-mon[52881]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:35.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:35 smithi192 ceph-mon[52881]: failed to encode map e130 with expected crc 2024-02-10T05:51:35.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:35 smithi192 ceph-mon[52881]: pgmap v211: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 14 peering, 11 active+undersized+degraded, 11 active+undersized, 60 active+clean; 326 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 5.1 MiB/s wr, 740 op/s; 142/1569 objects degraded (9.050%); 0 B/s, 72 keys/s, 0 objects/s recovering 2024-02-10T05:51:35.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:35 smithi192 ceph-mon[52881]: osdmap e131: 8 total, 8 up, 8 in 2024-02-10T05:51:35.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:35 smithi192 ceph-mon[52881]: failed to encode map e131 with expected crc 2024-02-10T05:51:36.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 259510 9.99 MB/sec execute 510 sec latency 273.378 ms 2024-02-10T05:51:36.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:36 smithi028 ceph-mon[66336]: failed to encode map e131 with expected crc 2024-02-10T05:51:36.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:36 smithi028 ceph-mon[66336]: failed to encode map e131 with expected crc 2024-02-10T05:51:36.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:36 smithi028 ceph-mon[66336]: osdmap e132: 8 total, 8 up, 8 in 2024-02-10T05:51:36.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:36 smithi028 ceph-mon[66336]: failed to encode map e132 with expected crc 2024-02-10T05:51:36.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:36 smithi028 ceph-mon[66336]: failed to encode map e132 with expected crc 2024-02-10T05:51:36.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:36 smithi028 ceph-mon[66336]: failed to encode map e132 with expected crc 2024-02-10T05:51:36.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:36 smithi192 ceph-mon[52881]: failed to encode map e131 with expected crc 2024-02-10T05:51:36.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:36 smithi192 ceph-mon[52881]: failed to encode map e131 with expected crc 2024-02-10T05:51:36.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:36 smithi192 ceph-mon[52881]: osdmap e132: 8 total, 8 up, 8 in 2024-02-10T05:51:36.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:36 smithi192 ceph-mon[52881]: failed to encode map e132 with expected crc 2024-02-10T05:51:36.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:36 smithi192 ceph-mon[52881]: failed to encode map e132 with expected crc 2024-02-10T05:51:36.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:36 smithi192 ceph-mon[52881]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 259575 9.98 MB/sec execute 511 sec latency 1010.033 ms 2024-02-10T05:51:37.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:37 smithi028 ceph-mon[66336]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:37 smithi028 ceph-mon[66336]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:37 smithi028 ceph-mon[66336]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:37 smithi028 ceph-mon[66336]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:37 smithi028 ceph-mon[66336]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:37 smithi028 ceph-mon[66336]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:37 smithi028 ceph-mon[66336]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:37 smithi028 ceph-mon[66336]: pgmap v214: 97 pgs: 87 active+clean, 5 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped; 333 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.2 MiB/s wr, 338 op/s; 85/1608 objects degraded (5.286%); 1.1 MiB/s, 72 keys/s, 67 objects/s recovering 2024-02-10T05:51:37.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:37 smithi028 ceph-mon[66336]: failed to encode map e133 with expected crc 2024-02-10T05:51:37.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:37 smithi028 ceph-mon[66336]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T05:51:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:37 smithi192 ceph-mon[52881]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:37 smithi192 ceph-mon[52881]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:37 smithi192 ceph-mon[52881]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:37 smithi192 ceph-mon[52881]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:37 smithi192 ceph-mon[52881]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:37 smithi192 ceph-mon[52881]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:37 smithi192 ceph-mon[52881]: failed to encode map e132 with expected crc 2024-02-10T05:51:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:37 smithi192 ceph-mon[52881]: pgmap v214: 97 pgs: 87 active+clean, 5 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped; 333 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.2 MiB/s wr, 338 op/s; 85/1608 objects degraded (5.286%); 1.1 MiB/s, 72 keys/s, 67 objects/s recovering 2024-02-10T05:51:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:37 smithi192 ceph-mon[52881]: failed to encode map e133 with expected crc 2024-02-10T05:51:37.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:37 smithi192 ceph-mon[52881]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T05:51:38.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 259843 9.97 MB/sec execute 512 sec latency 46.297 ms 2024-02-10T05:51:38.454 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:38 smithi028 ceph-mon[66336]: failed to encode map e133 with expected crc 2024-02-10T05:51:38.454 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:38 smithi028 ceph-mon[66336]: failed to encode map e133 with expected crc 2024-02-10T05:51:38.455 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:38 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 85/1608 objects degraded (5.286%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:38.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:38 smithi192 ceph-mon[52881]: failed to encode map e133 with expected crc 2024-02-10T05:51:38.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:38 smithi192 ceph-mon[52881]: failed to encode map e133 with expected crc 2024-02-10T05:51:38.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:38 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 85/1608 objects degraded (5.286%), 4 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 260065 9.97 MB/sec execute 513 sec latency 45.743 ms 2024-02-10T05:51:39.224 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:39 smithi028 ceph-mon[66336]: pgmap v216: 97 pgs: 87 active+clean, 5 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped; 333 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.9 MiB/s wr, 312 op/s; 85/1608 objects degraded (5.286%); 1.0 MiB/s, 64 objects/s recovering 2024-02-10T05:51:39.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:39 smithi192 ceph-mon[52881]: pgmap v216: 97 pgs: 87 active+clean, 5 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped; 333 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.9 MiB/s wr, 312 op/s; 85/1608 objects degraded (5.286%); 1.0 MiB/s, 64 objects/s recovering 2024-02-10T05:51:40.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 260403 9.95 MB/sec execute 514 sec latency 43.539 ms 2024-02-10T05:51:41.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 260679 9.94 MB/sec execute 515 sec latency 45.665 ms 2024-02-10T05:51:41.293 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:41 smithi028 ceph-mon[66336]: pgmap v217: 97 pgs: 91 active+clean, 4 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 333 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.2 MiB/s wr, 237 op/s; 41/1632 objects degraded (2.512%); 1.4 MiB/s, 53 objects/s recovering 2024-02-10T05:51:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:41 smithi192 ceph-mon[52881]: pgmap v217: 97 pgs: 91 active+clean, 4 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped; 333 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.2 MiB/s wr, 237 op/s; 41/1632 objects degraded (2.512%); 1.4 MiB/s, 53 objects/s recovering 2024-02-10T05:51:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: 5 261014 9.95 MB/sec execute 516 sec latency 51.689 ms 2024-02-10T05:51:42.349 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:42 smithi028 ceph-mon[66336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/1632 objects degraded (2.512%), 2 pgs degraded) 2024-02-10T05:51:42.349 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:42 smithi028 ceph-mon[66336]: Cluster is now healthy 2024-02-10T05:51:42.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:42 smithi192 ceph-mon[52881]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/1632 objects degraded (2.512%), 2 pgs degraded) 2024-02-10T05:51:42.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:42 smithi192 ceph-mon[52881]: Cluster is now healthy 2024-02-10T05:51:43.175 INFO:teuthology.orchestra.run.smithi028.stdout: 5 261360 9.94 MB/sec execute 517 sec latency 84.858 ms 2024-02-10T05:51:43.384 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:43 smithi028 ceph-mon[66336]: pgmap v218: 97 pgs: 97 active+clean; 335 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 5.1 MiB/s wr, 445 op/s; 1.3 MiB/s, 47 objects/s recovering 2024-02-10T05:51:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:43 smithi192 ceph-mon[52881]: pgmap v218: 97 pgs: 97 active+clean; 335 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 5.1 MiB/s wr, 445 op/s; 1.3 MiB/s, 47 objects/s recovering 2024-02-10T05:51:44.175 INFO:teuthology.orchestra.run.smithi028.stdout: 5 261717 9.93 MB/sec execute 518 sec latency 58.627 ms 2024-02-10T05:51:45.175 INFO:teuthology.orchestra.run.smithi028.stdout: 5 261980 9.92 MB/sec execute 519 sec latency 60.204 ms 2024-02-10T05:51:45.397 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:45 smithi028 ceph-mon[66336]: pgmap v219: 97 pgs: 97 active+clean; 335 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.7 MiB/s wr, 370 op/s; 629 KiB/s, 34 objects/s recovering 2024-02-10T05:51:45.397 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T05:51:45.397 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:45.397 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:51:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:45 smithi192 ceph-mon[52881]: pgmap v219: 97 pgs: 97 active+clean; 335 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.7 MiB/s wr, 370 op/s; 629 KiB/s, 34 objects/s recovering 2024-02-10T05:51:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T05:51:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:51:46.175 INFO:teuthology.orchestra.run.smithi028.stdout: 5 262294 9.92 MB/sec execute 520 sec latency 44.727 ms 2024-02-10T05:51:46.446 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:46 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T05:51:46.447 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:46 smithi028 ceph-mon[66336]: Upgrade: osd.5 is safe to restart 2024-02-10T05:51:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:46 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-10T05:51:46.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:46 smithi192 ceph-mon[52881]: Upgrade: osd.5 is safe to restart 2024-02-10T05:51:47.175 INFO:teuthology.orchestra.run.smithi028.stdout: 5 262757 9.93 MB/sec execute 521 sec latency 40.464 ms 2024-02-10T05:51:47.473 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:47 smithi028 ceph-mon[66336]: pgmap v220: 97 pgs: 97 active+clean; 320 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.9 MiB/s wr, 430 op/s; 512 KiB/s, 10 objects/s recovering 2024-02-10T05:51:47.473 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:47 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:47.473 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:47 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:51:47.473 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:47 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:47 smithi192 ceph-mon[52881]: pgmap v220: 97 pgs: 97 active+clean; 320 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.9 MiB/s wr, 430 op/s; 512 KiB/s, 10 objects/s recovering 2024-02-10T05:51:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:47 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:47 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:51:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:47 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:48.175 INFO:teuthology.orchestra.run.smithi028.stdout: 5 263232 9.96 MB/sec execute 522 sec latency 38.254 ms 2024-02-10T05:51:48.485 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:48 smithi028 ceph-mon[66336]: Upgrade: Updating osd.5 2024-02-10T05:51:48.485 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:48 smithi028 ceph-mon[66336]: Deploying daemon osd.5 on smithi192 2024-02-10T05:51:48.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:48 smithi192 ceph-mon[52881]: Upgrade: Updating osd.5 2024-02-10T05:51:48.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:48 smithi192 ceph-mon[52881]: Deploying daemon osd.5 on smithi192 2024-02-10T05:51:49.175 INFO:teuthology.orchestra.run.smithi028.stdout: 5 263542 9.96 MB/sec execute 523 sec latency 31.389 ms 2024-02-10T05:51:49.500 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:49 smithi028 ceph-mon[66336]: pgmap v221: 97 pgs: 97 active+clean; 320 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.5 MiB/s wr, 386 op/s; 460 KiB/s, 9 objects/s recovering 2024-02-10T05:51:49.510 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:49 smithi192 ceph-mon[52881]: pgmap v221: 97 pgs: 97 active+clean; 320 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.5 MiB/s wr, 386 op/s; 460 KiB/s, 9 objects/s recovering 2024-02-10T05:51:50.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 263815 9.94 MB/sec execute 524 sec latency 48.838 ms 2024-02-10T05:51:50.894 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:50 smithi192 systemd[1]: Stopping Ceph osd.5 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:51:51.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 264049 9.93 MB/sec execute 525 sec latency 55.989 ms 2024-02-10T05:51:51.299 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:51 smithi028 ceph-mon[66336]: pgmap v222: 97 pgs: 97 active+clean; 323 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.7 MiB/s wr, 376 op/s; 427 KiB/s, 8 objects/s recovering 2024-02-10T05:51:51.299 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:51 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:51.299 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:51 smithi028 ceph-mon[66336]: osd.5 marked itself down and dead 2024-02-10T05:51:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:51 smithi192 ceph-mon[52881]: pgmap v222: 97 pgs: 97 active+clean; 323 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.7 MiB/s wr, 376 op/s; 427 KiB/s, 8 objects/s recovering 2024-02-10T05:51:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:51 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:51 smithi192 ceph-mon[52881]: osd.5 marked itself down and dead 2024-02-10T05:51:51.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:50 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[34048]: 2024-02-10T05:51:50.892+0000 7f441e612700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:51:51.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:50 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[34048]: 2024-02-10T05:51:50.892+0000 7f441e612700 -1 osd.5 133 *** Got signal Terminated *** 2024-02-10T05:51:51.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:50 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[34048]: 2024-02-10T05:51:50.892+0000 7f441e612700 -1 osd.5 133 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:51:52.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 264191 9.91 MB/sec execute 526 sec latency 897.528 ms 2024-02-10T05:51:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:52 smithi192 ceph-mon[52881]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:51:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:52 smithi192 ceph-mon[52881]: failed to encode map e134 with expected crc 2024-02-10T05:51:52.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:52 smithi192 ceph-mon[52881]: failed to encode map e134 with expected crc 2024-02-10T05:51:52.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:52 smithi192 ceph-mon[52881]: osdmap e134: 8 total, 7 up, 8 in 2024-02-10T05:51:52.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:51 smithi192 podman[59651]: 2024-02-10 05:51:51.939531932 +0000 UTC m=+1.176857685 container died ae5279672ff834371b264fa41aed2c1ac278a7d39f3cf3103e7acefe8e9eacfb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., architecture=x86_64, ceph=True, com.redhat.component=centos-stream-container, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, version=8, io.buildah.version=1.19.8, io.openshift.expose-services=, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0) 2024-02-10T05:51:52.374 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:52 smithi028 ceph-mon[66336]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:51:52.375 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:52 smithi028 ceph-mon[66336]: failed to encode map e134 with expected crc 2024-02-10T05:51:52.375 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:52 smithi028 ceph-mon[66336]: failed to encode map e134 with expected crc 2024-02-10T05:51:52.375 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:52 smithi028 ceph-mon[66336]: osdmap e134: 8 total, 7 up, 8 in 2024-02-10T05:51:52.669 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:52 smithi192 podman[59651]: 2024-02-10 05:51:52.450839926 +0000 UTC m=+1.688165683 container cleanup ae5279672ff834371b264fa41aed2c1ac278a7d39f3cf3103e7acefe8e9eacfb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5, distribution-scope=public, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., name=centos-stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, ceph=True, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, version=8, RELEASE=HEAD, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-10T05:51:52.669 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:52 smithi192 bash[59651]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5 2024-02-10T05:51:53.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:52 smithi192 podman[59665]: 2024-02-10 05:51:52.669301398 +0000 UTC m=+0.731998819 container remove ae5279672ff834371b264fa41aed2c1ac278a7d39f3cf3103e7acefe8e9eacfb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5, vcs-type=git, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, ceph=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-10T05:51:53.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 264406 9.91 MB/sec execute 527 sec latency 1031.806 ms 2024-02-10T05:51:53.428 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:53 smithi028 ceph-mon[66336]: pgmap v224: 97 pgs: 13 stale+active+clean, 84 active+clean; 332 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.8 MiB/s wr, 438 op/s 2024-02-10T05:51:53.429 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:53 smithi028 ceph-mon[66336]: failed to encode map e135 with expected crc 2024-02-10T05:51:53.429 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:53 smithi028 ceph-mon[66336]: osdmap e135: 8 total, 7 up, 8 in 2024-02-10T05:51:53.429 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:53 smithi028 ceph-mon[66336]: failed to encode map e135 with expected crc 2024-02-10T05:51:53.429 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:53 smithi028 ceph-mon[66336]: failed to encode map e135 with expected crc 2024-02-10T05:51:53.429 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:53 smithi028 ceph-mon[66336]: failed to encode map e135 with expected crc 2024-02-10T05:51:53.430 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:53 smithi028 ceph-mon[66336]: failed to encode map e135 with expected crc 2024-02-10T05:51:53.430 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:53 smithi028 ceph-mon[66336]: failed to encode map e135 with expected crc 2024-02-10T05:51:53.514 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:53 smithi192 ceph-mon[52881]: pgmap v224: 97 pgs: 13 stale+active+clean, 84 active+clean; 332 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.8 MiB/s wr, 438 op/s 2024-02-10T05:51:53.514 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:53 smithi192 ceph-mon[52881]: failed to encode map e135 with expected crc 2024-02-10T05:51:53.515 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:53 smithi192 ceph-mon[52881]: osdmap e135: 8 total, 7 up, 8 in 2024-02-10T05:51:53.515 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:53 smithi192 ceph-mon[52881]: failed to encode map e135 with expected crc 2024-02-10T05:51:53.515 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:53 smithi192 ceph-mon[52881]: failed to encode map e135 with expected crc 2024-02-10T05:51:53.515 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:53 smithi192 ceph-mon[52881]: failed to encode map e135 with expected crc 2024-02-10T05:51:53.515 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:53 smithi192 ceph-mon[52881]: failed to encode map e135 with expected crc 2024-02-10T05:51:53.515 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:53 smithi192 ceph-mon[52881]: failed to encode map e135 with expected crc 2024-02-10T05:51:53.515 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:53 smithi192 podman[59755]: 2024-02-10 05:51:53.10501442 +0000 UTC m=+0.019858155 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:51:53.515 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:53 smithi192 podman[59755]: 2024-02-10 05:51:53.253721285 +0000 UTC m=+0.168565011 container create 088afd897170c4365bf23fc5a04e8910217194ca2dc549396df27ab7c53f9331 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=) 2024-02-10T05:51:53.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:53 smithi192 podman[59755]: 2024-02-10 05:51:53.514897229 +0000 UTC m=+0.429740951 container init 088afd897170c4365bf23fc5a04e8910217194ca2dc549396df27ab7c53f9331 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-10T05:51:53.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:53 smithi192 podman[59755]: 2024-02-10 05:51:53.518810809 +0000 UTC m=+0.433654537 container start 088afd897170c4365bf23fc5a04e8910217194ca2dc549396df27ab7c53f9331 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, ceph=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-10T05:51:53.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:53 smithi192 podman[59755]: 2024-02-10 05:51:53.615121041 +0000 UTC m=+0.529964775 container attach 088afd897170c4365bf23fc5a04e8910217194ca2dc549396df27ab7c53f9331 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-10T05:51:54.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 264707 9.90 MB/sec execute 528 sec latency 105.265 ms 2024-02-10T05:51:54.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:53 smithi192 podman[59755]: 2024-02-10 05:51:53.878373439 +0000 UTC m=+0.793217170 container died 088afd897170c4365bf23fc5a04e8910217194ca2dc549396df27ab7c53f9331 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-10T05:51:54.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:54 smithi028 ceph-mon[66336]: pgmap v226: 97 pgs: 2 active+undersized, 3 active+undersized+degraded, 12 stale+active+clean, 80 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.3 MiB/s wr, 365 op/s; 16/1689 objects degraded (0.947%) 2024-02-10T05:51:54.435 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:54 smithi028 ceph-mon[66336]: Health check failed: Degraded data redundancy: 16/1689 objects degraded (0.947%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:54.787 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:54 smithi192 ceph-mon[52881]: pgmap v226: 97 pgs: 2 active+undersized, 3 active+undersized+degraded, 12 stale+active+clean, 80 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.3 MiB/s wr, 365 op/s; 16/1689 objects degraded (0.947%) 2024-02-10T05:51:54.787 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:54 smithi192 ceph-mon[52881]: Health check failed: Degraded data redundancy: 16/1689 objects degraded (0.947%), 3 pgs degraded (PG_DEGRADED) 2024-02-10T05:51:54.787 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:54 smithi192 podman[59780]: 2024-02-10 05:51:54.520213385 +0000 UTC m=+0.632633026 container remove 088afd897170c4365bf23fc5a04e8910217194ca2dc549396df27ab7c53f9331 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-10T05:51:54.787 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:54 smithi192 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.5.service: Deactivated successfully. 2024-02-10T05:51:54.787 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:54 smithi192 systemd[1]: Stopped Ceph osd.5 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:51:54.787 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:54 smithi192 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.5.service: Consumed 5min 38.639s CPU time. 2024-02-10T05:51:55.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:54 smithi192 systemd[1]: Starting Ceph osd.5 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:51:55.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:55 smithi192 podman[59879]: 2024-02-10 05:51:54.930220869 +0000 UTC m=+0.023847265 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:51:55.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 265134 9.90 MB/sec execute 529 sec latency 58.279 ms 2024-02-10T05:51:55.348 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:55 smithi192 podman[59879]: 2024-02-10 05:51:55.230111791 +0000 UTC m=+0.323738176 container create 2ac9308fbfdd46dcb3a827dc42ccf6ac46f14b454c96e730113665f6f0aff2ce (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-290212a) 2024-02-10T05:51:55.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:55 smithi192 podman[59879]: 2024-02-10 05:51:55.458115711 +0000 UTC m=+0.551742101 container init 2ac9308fbfdd46dcb3a827dc42ccf6ac46f14b454c96e730113665f6f0aff2ce (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2) 2024-02-10T05:51:55.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:55 smithi192 podman[59879]: 2024-02-10 05:51:55.462624315 +0000 UTC m=+0.556250704 container start 2ac9308fbfdd46dcb3a827dc42ccf6ac46f14b454c96e730113665f6f0aff2ce (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-10T05:51:55.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:55 smithi192 podman[59879]: 2024-02-10 05:51:55.549936597 +0000 UTC m=+0.643562992 container attach 2ac9308fbfdd46dcb3a827dc42ccf6ac46f14b454c96e730113665f6f0aff2ce (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, RELEASE=main-290212a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:51:56.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 265478 9.89 MB/sec execute 530 sec latency 57.269 ms 2024-02-10T05:51:56.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:55 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate[59893]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T05:51:56.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate[59893]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T05:51:56.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate[59893]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T05:51:56.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate[59893]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T05:51:56.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate[59893]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-10T05:51:56.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 bash[59879]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T05:51:56.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 bash[59879]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T05:51:56.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 bash[59879]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T05:51:56.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 bash[59879]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T05:51:56.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 bash[59879]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-10T05:51:56.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 bash[59879]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T05:51:56.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 bash[59879]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-10T05:51:56.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate[59893]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T05:51:56.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate[59893]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-10T05:51:56.323 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 podman[59879]: 2024-02-10 05:51:56.033971117 +0000 UTC m=+1.127597508 container died 2ac9308fbfdd46dcb3a827dc42ccf6ac46f14b454c96e730113665f6f0aff2ce (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-290212a) 2024-02-10T05:51:57.055 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:56 smithi192 podman[60015]: 2024-02-10 05:51:56.796624087 +0000 UTC m=+0.755770764 container remove 2ac9308fbfdd46dcb3a827dc42ccf6ac46f14b454c96e730113665f6f0aff2ce (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-10T05:51:57.056 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:57 smithi192 podman[60080]: 2024-02-10 05:51:56.920339554 +0000 UTC m=+0.020886415 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:51:57.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 265829 9.90 MB/sec execute 531 sec latency 37.026 ms 2024-02-10T05:51:57.262 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:57 smithi028 ceph-mon[66336]: pgmap v227: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 322 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.0 MiB/s wr, 591 op/s; 178/1548 objects degraded (11.499%) 2024-02-10T05:51:57.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:57 smithi192 ceph-mon[52881]: pgmap v227: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 322 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.0 MiB/s wr, 591 op/s; 178/1548 objects degraded (11.499%) 2024-02-10T05:51:57.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:57 smithi192 podman[60080]: 2024-02-10 05:51:57.056012011 +0000 UTC m=+0.156558840 container create 000913de61d6317cefe868ee1de23cbeaaf84f4ad2f7b5358e36f3ad51ff18ba (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-10T05:51:57.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:57 smithi192 podman[60080]: 2024-02-10 05:51:57.275331915 +0000 UTC m=+0.375878745 container init 000913de61d6317cefe868ee1de23cbeaaf84f4ad2f7b5358e36f3ad51ff18ba (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:51:57.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:57 smithi192 podman[60080]: 2024-02-10 05:51:57.279799642 +0000 UTC m=+0.380346474 container start 000913de61d6317cefe868ee1de23cbeaaf84f4ad2f7b5358e36f3ad51ff18ba (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2) 2024-02-10T05:51:57.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:57 smithi192 ceph-osd[60099]: -- 172.21.15.192:0/3075010645 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560365fb31e0 con 0x560365f85000 2024-02-10T05:51:57.821 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:57 smithi192 bash[60080]: 000913de61d6317cefe868ee1de23cbeaaf84f4ad2f7b5358e36f3ad51ff18ba 2024-02-10T05:51:57.821 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:57 smithi192 systemd[1]: Started Ceph osd.5 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:51:58.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:51:57 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:51:57.820+0000 7f31c4fe47c0 -1 Falling back to public interface 2024-02-10T05:51:58.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 266182 9.90 MB/sec execute 532 sec latency 39.345 ms 2024-02-10T05:51:58.551 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:58 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:58.551 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:58 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:58.552 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:58 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:58.552 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:51:58 smithi028 ceph-mon[66336]: pgmap v228: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 322 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.2 MiB/s wr, 564 op/s; 178/1548 objects degraded (11.499%) 2024-02-10T05:51:58.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:58 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:58.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:58 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:51:58.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:58 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:58.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:51:58 smithi192 ceph-mon[52881]: pgmap v228: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 322 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.2 MiB/s wr, 564 op/s; 178/1548 objects degraded (11.499%) 2024-02-10T05:51:59.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 266561 9.91 MB/sec execute 533 sec latency 48.076 ms 2024-02-10T05:52:00.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 266867 9.92 MB/sec execute 534 sec latency 48.480 ms 2024-02-10T05:52:01.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:01 smithi028 ceph-mon[66336]: pgmap v229: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 328 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.1 MiB/s wr, 399 op/s; 191/1623 objects degraded (11.768%) 2024-02-10T05:52:01.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:01.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:52:01.076 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:01 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 191/1623 objects degraded (11.768%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:01.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 267119 9.91 MB/sec execute 535 sec latency 56.623 ms 2024-02-10T05:52:01.305 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:01 smithi192 ceph-mon[52881]: pgmap v229: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 328 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.1 MiB/s wr, 399 op/s; 191/1623 objects degraded (11.768%) 2024-02-10T05:52:01.305 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:01.305 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:52:01.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:01 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 191/1623 objects degraded (11.768%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 267387 9.90 MB/sec execute 536 sec latency 45.002 ms 2024-02-10T05:52:02.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:02 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:02.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:02 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:02.888 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:02 smithi028 ceph-mon[66336]: pgmap v230: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 332 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 447 op/s; 197/1689 objects degraded (11.664%) 2024-02-10T05:52:03.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:02 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:03.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:02 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:03.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:02 smithi192 ceph-mon[52881]: pgmap v230: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 332 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 447 op/s; 197/1689 objects degraded (11.664%) 2024-02-10T05:52:03.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 267668 9.89 MB/sec execute 537 sec latency 45.163 ms 2024-02-10T05:52:04.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 267950 9.89 MB/sec execute 538 sec latency 47.675 ms 2024-02-10T05:52:04.186 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:04.186 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:04.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:04.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:04.501 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:52:05.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 268252 9.88 MB/sec execute 539 sec latency 57.028 ms 2024-02-10T05:52:05.204 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:05 smithi028 ceph-mon[66336]: pgmap v231: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.3 MiB/s wr, 441 op/s; 196/1710 objects degraded (11.462%) 2024-02-10T05:52:05.460 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:05 smithi192 ceph-mon[52881]: pgmap v231: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.3 MiB/s wr, 441 op/s; 196/1710 objects degraded (11.462%) 2024-02-10T05:52:05.581 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (8m) 65s ago 17m 15.4M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (4m) 65s ago 17m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (4m) 3s ago 16m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (7m) 65s ago 17m 45.8M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (11m) 65s ago 11m 38.5M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (11m) 3s ago 11m 15.1M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (7m) 65s ago 18m 504M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (6m) 3s ago 16m 453M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (5m) 65s ago 18m 51.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (5m) 3s ago 16m 42.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (11m) 65s ago 11m 282M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (11m) 3s ago 11m 86.2M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (7m) 65s ago 17m 13.5M - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:52:05.582 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (7m) 3s ago 16m 21.4M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:52:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (3m) 65s ago 15m 339M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:52:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (2m) 65s ago 15m 289M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:52:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (2m) 65s ago 14m 233M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 a45f0e24ae44 2024-02-10T05:52:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (75s) 65s ago 14m 11.6M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 25f4cf94969e 2024-02-10T05:52:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (46s) 3s ago 14m 269M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 b15b2733ec77 2024-02-10T05:52:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (8s) 3s ago 13m 12.6M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 000913de61d6 2024-02-10T05:52:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (13m) 3s ago 13m 555M 2704M 17.2.0 e1d6a67b021e d9ae5a38d430 2024-02-10T05:52:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (12m) 3s ago 12m 601M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:52:05.583 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (6m) 65s ago 16m 62.1M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:52:05.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:52:05 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:52:05.635+0000 7f31c4fe47c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-02-10T05:52:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 268597 9.87 MB/sec execute 540 sec latency 98.602 ms 2024-02-10T05:52:06.212 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:06 smithi028 ceph-mon[66336]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:06.212 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:06 smithi028 ceph-mon[66336]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:06.291 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:52:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:52:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:52:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:52:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:52:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:52:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:52:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:52:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-10T05:52:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 5 2024-02-10T05:52:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:52:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:52:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:52:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:52:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:52:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-10T05:52:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 9 2024-02-10T05:52:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:52:06.292 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:52:06.491 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:06 smithi192 ceph-mon[52881]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:06.492 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:06 smithi192 ceph-mon[52881]: from='client.34384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:06.837 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:52:06.837 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:52:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:52:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:52:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:52:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:52:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:52:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:52:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:52:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "12/23 daemons upgraded", 2024-02-10T05:52:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T05:52:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:52:06.838 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:52:07.122 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:52:06 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:52:06.917+0000 7f31c4fe47c0 -1 osd.5 133 log_to_monitors true 2024-02-10T05:52:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: 5 268927 9.86 MB/sec execute 541 sec latency 71.905 ms 2024-02-10T05:52:07.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='client.44323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:07.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: pgmap v232: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 325 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.1 MiB/s wr, 520 op/s; 178/1578 objects degraded (11.280%) 2024-02-10T05:52:07.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 178/1578 objects degraded (11.280%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:07.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/3649381091' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:07.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:07.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:07.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:07.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:07.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:07.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='osd.5 [v2:172.21.15.192:6808/2316483711,v1:172.21.15.192:6809/2316483711]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:52:07.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:52:07.229 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:07.230 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:07.230 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:07.230 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:07.230 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:07.230 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:07 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T05:52:07.449 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 178/1578 objects degraded (11.280%), 23 pgs degraded 2024-02-10T05:52:07.449 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-10T05:52:07.449 INFO:teuthology.orchestra.run.smithi028.stdout: osd.5 (root=default,host=smithi192) is down 2024-02-10T05:52:07.449 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 178/1578 objects degraded (11.280%), 23 pgs degraded 2024-02-10T05:52:07.449 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-02-10T05:52:07.449 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.3 is active+undersized+degraded, acting [2,7] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.b is active+undersized+degraded, acting [7,4] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.18 is active+undersized+degraded, acting [4,7] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1c is active+undersized+degraded, acting [7,2] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1e is active+undersized+degraded, acting [3,0] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-02-10T05:52:07.450 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-02-10T05:52:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-02-10T05:52:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-02-10T05:52:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-02-10T05:52:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-02-10T05:52:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-02-10T05:52:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-02-10T05:52:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-02-10T05:52:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-02-10T05:52:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='client.44323 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: pgmap v232: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 325 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.1 MiB/s wr, 520 op/s; 178/1578 objects degraded (11.280%) 2024-02-10T05:52:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 178/1578 objects degraded (11.280%), 23 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/3649381091' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:07.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:07.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='osd.5 [v2:172.21.15.192:6808/2316483711,v1:172.21.15.192:6809/2316483711]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:52:07.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:52:07.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:07.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:07.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:07.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:07.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:07.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:07 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T05:52:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 269219 9.86 MB/sec execute 542 sec latency 175.849 ms 2024-02-10T05:52:08.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:08 smithi028 ceph-mon[66336]: from='client.44331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:08.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:08 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T05:52:08.280 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:08 smithi028 ceph-mon[66336]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-10T05:52:08.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:08 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/3902228668' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:52:08.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:08 smithi028 ceph-mon[66336]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:52:08.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:08 smithi028 ceph-mon[66336]: osdmap e136: 8 total, 7 up, 8 in 2024-02-10T05:52:08.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:08 smithi028 ceph-mon[66336]: from='osd.5 [v2:172.21.15.192:6808/2316483711,v1:172.21.15.192:6809/2316483711]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:52:08.281 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:08 smithi028 ceph-mon[66336]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:52:08.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:52:08 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:52:08.417+0000 7f31b2434700 -1 osd.5 133 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:52:08.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:08 smithi192 ceph-mon[52881]: from='client.44331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:08.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:08 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T05:52:08.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:08 smithi192 ceph-mon[52881]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-10T05:52:08.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:08 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/3902228668' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:52:08.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:08 smithi192 ceph-mon[52881]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:52:08.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:08 smithi192 ceph-mon[52881]: osdmap e136: 8 total, 7 up, 8 in 2024-02-10T05:52:08.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:08 smithi192 ceph-mon[52881]: from='osd.5 [v2:172.21.15.192:6808/2316483711,v1:172.21.15.192:6809/2316483711]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:52:08.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:08 smithi192 ceph-mon[52881]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:52:09.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 269493 9.86 MB/sec execute 543 sec latency 226.169 ms 2024-02-10T05:52:09.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:09 smithi028 ceph-mon[66336]: failed to encode map e136 with expected crc 2024-02-10T05:52:09.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:09 smithi028 ceph-mon[66336]: pgmap v234: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 325 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 443 op/s; 178/1578 objects degraded (11.280%) 2024-02-10T05:52:09.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:09 smithi028 ceph-mon[66336]: from='osd.5 ' entity='osd.5' 2024-02-10T05:52:09.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:09 smithi192 ceph-mon[52881]: failed to encode map e136 with expected crc 2024-02-10T05:52:09.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:09 smithi192 ceph-mon[52881]: pgmap v234: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 325 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 443 op/s; 178/1578 objects degraded (11.280%) 2024-02-10T05:52:09.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:09 smithi192 ceph-mon[52881]: from='osd.5 ' entity='osd.5' 2024-02-10T05:52:10.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 269778 9.86 MB/sec execute 544 sec latency 699.889 ms 2024-02-10T05:52:10.518 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:10 smithi028 ceph-mon[66336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:52:10.519 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:10 smithi028 ceph-mon[66336]: failed to encode map e137 with expected crc 2024-02-10T05:52:10.519 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:10 smithi028 ceph-mon[66336]: osd.5 [v2:172.21.15.192:6808/2316483711,v1:172.21.15.192:6809/2316483711] boot 2024-02-10T05:52:10.519 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:10 smithi028 ceph-mon[66336]: osdmap e137: 8 total, 8 up, 8 in 2024-02-10T05:52:10.519 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:52:10.519 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:10 smithi028 ceph-mon[66336]: failed to encode map e137 with expected crc 2024-02-10T05:52:10.519 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:10 smithi028 ceph-mon[66336]: failed to encode map e137 with expected crc 2024-02-10T05:52:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:10 smithi192 ceph-mon[52881]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:52:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:10 smithi192 ceph-mon[52881]: failed to encode map e137 with expected crc 2024-02-10T05:52:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:10 smithi192 ceph-mon[52881]: osd.5 [v2:172.21.15.192:6808/2316483711,v1:172.21.15.192:6809/2316483711] boot 2024-02-10T05:52:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:10 smithi192 ceph-mon[52881]: osdmap e137: 8 total, 8 up, 8 in 2024-02-10T05:52:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:52:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:10 smithi192 ceph-mon[52881]: failed to encode map e137 with expected crc 2024-02-10T05:52:10.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:10 smithi192 ceph-mon[52881]: failed to encode map e137 with expected crc 2024-02-10T05:52:11.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 269784 9.85 MB/sec execute 545 sec latency 1699.916 ms 2024-02-10T05:52:11.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:11 smithi028 ceph-mon[66336]: pgmap v236: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 320 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.1 MiB/s wr, 368 op/s; 181/1560 objects degraded (11.603%) 2024-02-10T05:52:11.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:11 smithi028 ceph-mon[66336]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:11 smithi028 ceph-mon[66336]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T05:52:11.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:11 smithi028 ceph-mon[66336]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:11 smithi028 ceph-mon[66336]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:11 smithi028 ceph-mon[66336]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:11 smithi028 ceph-mon[66336]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:11 smithi028 ceph-mon[66336]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:11 smithi028 ceph-mon[66336]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:11 smithi192 ceph-mon[52881]: pgmap v236: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 320 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.1 MiB/s wr, 368 op/s; 181/1560 objects degraded (11.603%) 2024-02-10T05:52:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:11 smithi192 ceph-mon[52881]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:11 smithi192 ceph-mon[52881]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T05:52:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:11 smithi192 ceph-mon[52881]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:11 smithi192 ceph-mon[52881]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:11 smithi192 ceph-mon[52881]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:11 smithi192 ceph-mon[52881]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:11 smithi192 ceph-mon[52881]: failed to encode map e138 with expected crc 2024-02-10T05:52:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:11 smithi192 ceph-mon[52881]: failed to encode map e138 with expected crc 2024-02-10T05:52:12.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 269977 9.84 MB/sec execute 546 sec latency 2350.553 ms 2024-02-10T05:52:12.579 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:12 smithi028 ceph-mon[66336]: failed to encode map e139 with expected crc 2024-02-10T05:52:12.580 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:12 smithi028 ceph-mon[66336]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T05:52:12.580 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:12 smithi028 ceph-mon[66336]: failed to encode map e139 with expected crc 2024-02-10T05:52:12.580 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:12 smithi028 ceph-mon[66336]: failed to encode map e139 with expected crc 2024-02-10T05:52:12.580 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:12 smithi028 ceph-mon[66336]: pgmap v239: 97 pgs: 3 remapped+peering, 6 peering, 6 active+undersized, 12 active+undersized+degraded, 70 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 5.2 MiB/s wr, 510 op/s; 108/1650 objects degraded (6.545%); 25 KiB/s, 0 objects/s recovering 2024-02-10T05:52:12.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:12 smithi192 ceph-mon[52881]: failed to encode map e139 with expected crc 2024-02-10T05:52:12.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:12 smithi192 ceph-mon[52881]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T05:52:12.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:12 smithi192 ceph-mon[52881]: failed to encode map e139 with expected crc 2024-02-10T05:52:12.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:12 smithi192 ceph-mon[52881]: failed to encode map e139 with expected crc 2024-02-10T05:52:12.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:12 smithi192 ceph-mon[52881]: pgmap v239: 97 pgs: 3 remapped+peering, 6 peering, 6 active+undersized, 12 active+undersized+degraded, 70 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 5.2 MiB/s wr, 510 op/s; 108/1650 objects degraded (6.545%); 25 KiB/s, 0 objects/s recovering 2024-02-10T05:52:13.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 270279 9.85 MB/sec execute 547 sec latency 472.699 ms 2024-02-10T05:52:13.594 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:13 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 108/1650 objects degraded (6.545%), 12 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:13.595 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:13 smithi028 ceph-mon[66336]: osdmap e140: 8 total, 8 up, 8 in 2024-02-10T05:52:13.595 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:13 smithi028 ceph-mon[66336]: failed to encode map e140 with expected crc 2024-02-10T05:52:13.595 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:13 smithi028 ceph-mon[66336]: failed to encode map e140 with expected crc 2024-02-10T05:52:13.595 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:13 smithi028 ceph-mon[66336]: failed to encode map e140 with expected crc 2024-02-10T05:52:13.595 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:13 smithi028 ceph-mon[66336]: failed to encode map e140 with expected crc 2024-02-10T05:52:13.595 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:13 smithi028 ceph-mon[66336]: failed to encode map e140 with expected crc 2024-02-10T05:52:13.595 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:13 smithi028 ceph-mon[66336]: failed to encode map e140 with expected crc 2024-02-10T05:52:13.595 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:13 smithi028 ceph-mon[66336]: osdmap e141: 8 total, 8 up, 8 in 2024-02-10T05:52:13.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:13 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 108/1650 objects degraded (6.545%), 12 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:13.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:13 smithi192 ceph-mon[52881]: osdmap e140: 8 total, 8 up, 8 in 2024-02-10T05:52:13.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:13 smithi192 ceph-mon[52881]: failed to encode map e140 with expected crc 2024-02-10T05:52:13.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:13 smithi192 ceph-mon[52881]: failed to encode map e140 with expected crc 2024-02-10T05:52:13.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:13 smithi192 ceph-mon[52881]: failed to encode map e140 with expected crc 2024-02-10T05:52:13.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:13 smithi192 ceph-mon[52881]: failed to encode map e140 with expected crc 2024-02-10T05:52:13.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:13 smithi192 ceph-mon[52881]: failed to encode map e140 with expected crc 2024-02-10T05:52:13.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:13 smithi192 ceph-mon[52881]: failed to encode map e140 with expected crc 2024-02-10T05:52:13.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:13 smithi192 ceph-mon[52881]: osdmap e141: 8 total, 8 up, 8 in 2024-02-10T05:52:14.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 270560 9.85 MB/sec execute 548 sec latency 465.934 ms 2024-02-10T05:52:14.618 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:14 smithi028 ceph-mon[66336]: failed to encode map e141 with expected crc 2024-02-10T05:52:14.619 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:14 smithi028 ceph-mon[66336]: failed to encode map e141 with expected crc 2024-02-10T05:52:14.619 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:14 smithi028 ceph-mon[66336]: failed to encode map e141 with expected crc 2024-02-10T05:52:14.619 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:14 smithi028 ceph-mon[66336]: failed to encode map e141 with expected crc 2024-02-10T05:52:14.619 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:14 smithi028 ceph-mon[66336]: failed to encode map e141 with expected crc 2024-02-10T05:52:14.619 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:14 smithi028 ceph-mon[66336]: failed to encode map e141 with expected crc 2024-02-10T05:52:14.619 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:14 smithi028 ceph-mon[66336]: osdmap e142: 8 total, 8 up, 8 in 2024-02-10T05:52:14.619 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:14 smithi028 ceph-mon[66336]: pgmap v243: 97 pgs: 3 remapped+peering, 7 peering, 1 active+undersized, 86 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.5 MiB/s wr, 82 op/s; 771 KiB/s, 47 objects/s recovering 2024-02-10T05:52:14.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:14 smithi192 ceph-mon[52881]: failed to encode map e141 with expected crc 2024-02-10T05:52:14.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:14 smithi192 ceph-mon[52881]: failed to encode map e141 with expected crc 2024-02-10T05:52:14.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:14 smithi192 ceph-mon[52881]: failed to encode map e141 with expected crc 2024-02-10T05:52:14.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:14 smithi192 ceph-mon[52881]: failed to encode map e141 with expected crc 2024-02-10T05:52:14.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:14 smithi192 ceph-mon[52881]: failed to encode map e141 with expected crc 2024-02-10T05:52:14.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:14 smithi192 ceph-mon[52881]: failed to encode map e141 with expected crc 2024-02-10T05:52:14.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:14 smithi192 ceph-mon[52881]: osdmap e142: 8 total, 8 up, 8 in 2024-02-10T05:52:14.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:14 smithi192 ceph-mon[52881]: pgmap v243: 97 pgs: 3 remapped+peering, 7 peering, 1 active+undersized, 86 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.5 MiB/s wr, 82 op/s; 771 KiB/s, 47 objects/s recovering 2024-02-10T05:52:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 270814 9.84 MB/sec execute 549 sec latency 59.210 ms 2024-02-10T05:52:15.645 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:15 smithi028 ceph-mon[66336]: failed to encode map e142 with expected crc 2024-02-10T05:52:15.646 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:15 smithi028 ceph-mon[66336]: failed to encode map e142 with expected crc 2024-02-10T05:52:15.646 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:15 smithi028 ceph-mon[66336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 108/1650 objects degraded (6.545%), 12 pgs degraded) 2024-02-10T05:52:15.646 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:15 smithi028 ceph-mon[66336]: Cluster is now healthy 2024-02-10T05:52:15.646 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:15.646 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:52:15.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:15 smithi192 ceph-mon[52881]: failed to encode map e142 with expected crc 2024-02-10T05:52:15.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:15 smithi192 ceph-mon[52881]: failed to encode map e142 with expected crc 2024-02-10T05:52:15.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:15 smithi192 ceph-mon[52881]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 108/1650 objects degraded (6.545%), 12 pgs degraded) 2024-02-10T05:52:15.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:15 smithi192 ceph-mon[52881]: Cluster is now healthy 2024-02-10T05:52:15.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:15.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:52:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 271126 9.83 MB/sec execute 550 sec latency 37.645 ms 2024-02-10T05:52:16.658 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:16 smithi028 ceph-mon[66336]: pgmap v244: 97 pgs: 1 peering, 96 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.1 MiB/s wr, 372 op/s; 2.6 MiB/s, 61 objects/s recovering 2024-02-10T05:52:16.821 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:16 smithi192 ceph-mon[52881]: pgmap v244: 97 pgs: 1 peering, 96 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.1 MiB/s wr, 372 op/s; 2.6 MiB/s, 61 objects/s recovering 2024-02-10T05:52:17.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 271557 9.83 MB/sec execute 551 sec latency 35.953 ms 2024-02-10T05:52:18.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 272013 9.83 MB/sec execute 552 sec latency 32.923 ms 2024-02-10T05:52:18.919 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:18 smithi028 ceph-mon[66336]: pgmap v245: 97 pgs: 1 peering, 96 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.3 MiB/s wr, 300 op/s; 2.0 MiB/s, 46 objects/s recovering 2024-02-10T05:52:19.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 272482 9.82 MB/sec execute 553 sec latency 44.921 ms 2024-02-10T05:52:19.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:18 smithi192 ceph-mon[52881]: pgmap v245: 97 pgs: 1 peering, 96 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.3 MiB/s wr, 300 op/s; 2.0 MiB/s, 46 objects/s recovering 2024-02-10T05:52:20.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 272849 9.82 MB/sec execute 554 sec latency 47.405 ms 2024-02-10T05:52:21.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 273250 9.84 MB/sec execute 555 sec latency 37.791 ms 2024-02-10T05:52:21.211 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:21 smithi028 ceph-mon[66336]: pgmap v246: 97 pgs: 97 active+clean; 335 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.6 MiB/s wr, 421 op/s; 1.5 MiB/s, 36 objects/s recovering 2024-02-10T05:52:21.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:21 smithi192 ceph-mon[52881]: pgmap v246: 97 pgs: 97 active+clean; 335 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.6 MiB/s wr, 421 op/s; 1.5 MiB/s, 36 objects/s recovering 2024-02-10T05:52:22.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 273657 9.85 MB/sec execute 556 sec latency 41.412 ms 2024-02-10T05:52:23.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 273993 9.85 MB/sec execute 557 sec latency 46.767 ms 2024-02-10T05:52:23.253 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:23 smithi028 ceph-mon[66336]: pgmap v247: 97 pgs: 97 active+clean; 322 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.5 MiB/s wr, 501 op/s; 1.3 MiB/s, 30 objects/s recovering 2024-02-10T05:52:23.253 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T05:52:23.254 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:23 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T05:52:23.254 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:23 smithi028 ceph-mon[66336]: Upgrade: osd.6 is safe to restart 2024-02-10T05:52:23.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:23 smithi192 ceph-mon[52881]: pgmap v247: 97 pgs: 97 active+clean; 322 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.5 MiB/s wr, 501 op/s; 1.3 MiB/s, 30 objects/s recovering 2024-02-10T05:52:23.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T05:52:23.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:23 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-10T05:52:23.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:23 smithi192 ceph-mon[52881]: Upgrade: osd.6 is safe to restart 2024-02-10T05:52:24.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 274240 9.85 MB/sec execute 558 sec latency 48.532 ms 2024-02-10T05:52:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 274523 9.83 MB/sec execute 559 sec latency 40.578 ms 2024-02-10T05:52:25.286 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:25 smithi028 ceph-mon[66336]: pgmap v248: 97 pgs: 97 active+clean; 324 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.3 MiB/s wr, 462 op/s; 920 KiB/s, 10 objects/s recovering 2024-02-10T05:52:25.287 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:25.287 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:52:25.287 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:25 smithi192 ceph-mon[52881]: pgmap v248: 97 pgs: 97 active+clean; 324 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.3 MiB/s wr, 462 op/s; 920 KiB/s, 10 objects/s recovering 2024-02-10T05:52:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:52:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:26.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 274779 9.82 MB/sec execute 560 sec latency 56.909 ms 2024-02-10T05:52:26.307 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:26 smithi028 ceph-mon[66336]: Upgrade: Updating osd.6 2024-02-10T05:52:26.307 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:26 smithi028 ceph-mon[66336]: Deploying daemon osd.6 on smithi192 2024-02-10T05:52:26.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:26 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:26.461 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:26 smithi192 ceph-mon[52881]: Upgrade: Updating osd.6 2024-02-10T05:52:26.461 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:26 smithi192 ceph-mon[52881]: Deploying daemon osd.6 on smithi192 2024-02-10T05:52:26.462 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:26 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:27.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 275060 9.82 MB/sec execute 561 sec latency 65.463 ms 2024-02-10T05:52:27.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:27 smithi192 ceph-mon[52881]: pgmap v249: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.8 MiB/s wr, 577 op/s; 778 KiB/s, 9 objects/s recovering 2024-02-10T05:52:27.349 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:27 smithi028 ceph-mon[66336]: pgmap v249: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.8 MiB/s wr, 577 op/s; 778 KiB/s, 9 objects/s recovering 2024-02-10T05:52:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 275399 9.82 MB/sec execute 562 sec latency 57.322 ms 2024-02-10T05:52:28.884 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:28 smithi192 systemd[1]: Stopping Ceph osd.6 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:52:29.128 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:28 smithi028 ceph-mon[66336]: pgmap v250: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 455 op/s 2024-02-10T05:52:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 275797 9.81 MB/sec execute 563 sec latency 52.308 ms 2024-02-10T05:52:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:28 smithi192 ceph-mon[52881]: pgmap v250: 97 pgs: 97 active+clean; 335 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.5 MiB/s wr, 455 op/s 2024-02-10T05:52:29.322 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:29 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[37111]: 2024-02-10T05:52:29.015+0000 7ff88c91a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:52:29.322 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:29 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[37111]: 2024-02-10T05:52:29.015+0000 7ff88c91a700 -1 osd.6 142 *** Got signal Terminated *** 2024-02-10T05:52:29.322 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:29 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[37111]: 2024-02-10T05:52:29.015+0000 7ff88c91a700 -1 osd.6 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:52:29.885 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:29 smithi028 ceph-mon[66336]: osd.6 marked itself down and dead 2024-02-10T05:52:30.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 276057 9.80 MB/sec execute 564 sec latency 305.652 ms 2024-02-10T05:52:30.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:29 smithi192 ceph-mon[52881]: osd.6 marked itself down and dead 2024-02-10T05:52:30.807 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:30 smithi192 podman[61916]: 2024-02-10 05:52:30.54118097 +0000 UTC m=+1.709240221 container died d9ae5a38d4305681c004881ad96bffb92b78fcc4f754106d3efaca741d334289 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, ceph=True, maintainer=Guillaume Abrioux , vcs-type=git, distribution-scope=public, io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-10T05:52:31.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:31 smithi192 podman[61916]: 2024-02-10 05:52:31.007224468 +0000 UTC m=+2.175283717 container cleanup d9ae5a38d4305681c004881ad96bffb92b78fcc4f754106d3efaca741d334289 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, ceph=True, name=centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , vcs-type=git, io.buildah.version=1.19.8, release=754, com.redhat.component=centos-stream-container, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, version=8, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-10T05:52:31.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:31 smithi192 bash[61916]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6 2024-02-10T05:52:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:52:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: osdmap e143: 8 total, 7 up, 8 in 2024-02-10T05:52:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: failed to encode map e143 with expected crc 2024-02-10T05:52:31.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: failed to encode map e143 with expected crc 2024-02-10T05:52:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: failed to encode map e143 with expected crc 2024-02-10T05:52:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 6]}]: dispatch 2024-02-10T05:52:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1", "id": [4, 0]}]: dispatch 2024-02-10T05:52:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 6]}]: dispatch 2024-02-10T05:52:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-02-10T05:52:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: pgmap v252: 97 pgs: 10 stale+active+clean, 87 active+clean; 331 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.8 MiB/s wr, 416 op/s 2024-02-10T05:52:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T05:52:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [4, 6]}]: dispatch 2024-02-10T05:52:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 6]}]: dispatch 2024-02-10T05:52:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 6]}]: dispatch 2024-02-10T05:52:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]: dispatch 2024-02-10T05:52:31.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 6]}]: dispatch 2024-02-10T05:52:31.074 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:31.074 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:52:31.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:52:31.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: osdmap e143: 8 total, 7 up, 8 in 2024-02-10T05:52:31.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: failed to encode map e143 with expected crc 2024-02-10T05:52:31.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: failed to encode map e143 with expected crc 2024-02-10T05:52:31.135 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: failed to encode map e143 with expected crc 2024-02-10T05:52:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 6]}]: dispatch 2024-02-10T05:52:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1", "id": [4, 0]}]: dispatch 2024-02-10T05:52:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 6]}]: dispatch 2024-02-10T05:52:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-02-10T05:52:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: pgmap v252: 97 pgs: 10 stale+active+clean, 87 active+clean; 331 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.8 MiB/s wr, 416 op/s 2024-02-10T05:52:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-10T05:52:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [4, 6]}]: dispatch 2024-02-10T05:52:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 6]}]: dispatch 2024-02-10T05:52:31.136 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 6]}]: dispatch 2024-02-10T05:52:31.137 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]: dispatch 2024-02-10T05:52:31.137 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 6]}]: dispatch 2024-02-10T05:52:31.137 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:31.137 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:52:31.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 276144 9.79 MB/sec execute 565 sec latency 1040.544 ms 2024-02-10T05:52:31.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:31 smithi192 podman[61932]: 2024-02-10 05:52:31.401593244 +0000 UTC m=+0.857780143 container remove d9ae5a38d4305681c004881ad96bffb92b78fcc4f754106d3efaca741d334289 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, ceph=True, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, version=8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-10T05:52:32.149 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 6]}]': finished 2024-02-10T05:52:32.149 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1", "id": [4, 0]}]': finished 2024-02-10T05:52:32.149 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 6]}]': finished 2024-02-10T05:52:32.149 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-02-10T05:52:32.149 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-10T05:52:32.149 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [4, 6]}]': finished 2024-02-10T05:52:32.149 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 6]}]': finished 2024-02-10T05:52:32.149 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 6]}]': finished 2024-02-10T05:52:32.150 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]': finished 2024-02-10T05:52:32.150 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 6]}]': finished 2024-02-10T05:52:32.150 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: failed to encode map e144 with expected crc 2024-02-10T05:52:32.150 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: osdmap e144: 8 total, 7 up, 8 in 2024-02-10T05:52:32.150 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:31 smithi028 ceph-mon[66336]: failed to encode map e144 with expected crc 2024-02-10T05:52:32.164 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:31 smithi192 podman[62025]: 2024-02-10 05:52:31.726357533 +0000 UTC m=+0.024422651 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:52:32.164 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:31 smithi192 podman[62025]: 2024-02-10 05:52:31.844670987 +0000 UTC m=+0.142736070 container create 8b5a66baffacd32c9ef1360425615e7d2eff0f2aaaccfa5e86e4a255d82f1c77 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-deactivate, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-10T05:52:32.164 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:32 smithi192 podman[62025]: 2024-02-10 05:52:32.065174702 +0000 UTC m=+0.363239797 container init 8b5a66baffacd32c9ef1360425615e7d2eff0f2aaaccfa5e86e4a255d82f1c77 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:52:32.164 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:32 smithi192 podman[62025]: 2024-02-10 05:52:32.070124183 +0000 UTC m=+0.368189277 container start 8b5a66baffacd32c9ef1360425615e7d2eff0f2aaaccfa5e86e4a255d82f1c77 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 6]}]': finished 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1", "id": [4, 0]}]': finished 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 6]}]': finished 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [4, 6]}]': finished 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 6]}]': finished 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [3, 6]}]': finished 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]': finished 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 6]}]': finished 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: failed to encode map e144 with expected crc 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: osdmap e144: 8 total, 7 up, 8 in 2024-02-10T05:52:32.165 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:31 smithi192 ceph-mon[52881]: failed to encode map e144 with expected crc 2024-02-10T05:52:32.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 276534 9.79 MB/sec execute 566 sec latency 49.465 ms 2024-02-10T05:52:32.572 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:32 smithi192 podman[62025]: 2024-02-10 05:52:32.164648883 +0000 UTC m=+0.462713983 container attach 8b5a66baffacd32c9ef1360425615e7d2eff0f2aaaccfa5e86e4a255d82f1c77 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:52:32.572 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:32 smithi192 podman[62025]: 2024-02-10 05:52:32.394042682 +0000 UTC m=+0.692107757 container died 8b5a66baffacd32c9ef1360425615e7d2eff0f2aaaccfa5e86e4a255d82f1c77 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-10T05:52:33.106 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:33 smithi192 podman[62051]: 2024-02-10 05:52:33.095154571 +0000 UTC m=+0.695308183 container remove 8b5a66baffacd32c9ef1360425615e7d2eff0f2aaaccfa5e86e4a255d82f1c77 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-290212a, io.buildah.version=1.33.2) 2024-02-10T05:52:33.107 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:32 smithi192 ceph-mon[52881]: failed to encode map e145 with expected crc 2024-02-10T05:52:33.107 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:32 smithi192 ceph-mon[52881]: osdmap e145: 8 total, 7 up, 8 in 2024-02-10T05:52:33.107 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:32 smithi192 ceph-mon[52881]: failed to encode map e145 with expected crc 2024-02-10T05:52:33.107 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:32 smithi192 ceph-mon[52881]: pgmap v255: 97 pgs: 5 active+undersized+degraded, 9 peering, 7 stale+active+clean, 76 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 341 B/s rd, 2.9 MiB/s wr, 380 op/s; 37/1509 objects degraded (2.452%) 2024-02-10T05:52:33.167 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:32 smithi028 ceph-mon[66336]: failed to encode map e145 with expected crc 2024-02-10T05:52:33.167 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:32 smithi028 ceph-mon[66336]: osdmap e145: 8 total, 7 up, 8 in 2024-02-10T05:52:33.167 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:32 smithi028 ceph-mon[66336]: failed to encode map e145 with expected crc 2024-02-10T05:52:33.168 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:32 smithi028 ceph-mon[66336]: pgmap v255: 97 pgs: 5 active+undersized+degraded, 9 peering, 7 stale+active+clean, 76 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 341 B/s rd, 2.9 MiB/s wr, 380 op/s; 37/1509 objects degraded (2.452%) 2024-02-10T05:52:33.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 276987 9.81 MB/sec execute 567 sec latency 35.407 ms 2024-02-10T05:52:33.566 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:33 smithi192 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.6.service: Deactivated successfully. 2024-02-10T05:52:33.566 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:33 smithi192 systemd[1]: Stopped Ceph osd.6 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:52:33.566 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:33 smithi192 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.6.service: Consumed 4min 54.781s CPU time. 2024-02-10T05:52:33.566 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:33 smithi192 systemd[1]: Starting Ceph osd.6 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:52:33.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:33 smithi192 podman[62154]: 2024-02-10 05:52:33.468508418 +0000 UTC m=+0.020236328 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:52:33.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:33 smithi192 podman[62154]: 2024-02-10 05:52:33.604795149 +0000 UTC m=+0.156523040 container create 86e48d87dc937ebfcd08597bf6546fd8fb70d056347d4091b841ddbc0aa4aaea (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-10T05:52:33.931 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:33 smithi028 ceph-mon[66336]: failed to encode map e146 with expected crc 2024-02-10T05:52:33.932 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:33 smithi028 ceph-mon[66336]: osdmap e146: 8 total, 7 up, 8 in 2024-02-10T05:52:33.932 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:33 smithi028 ceph-mon[66336]: failed to encode map e146 with expected crc 2024-02-10T05:52:33.932 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:33 smithi028 ceph-mon[66336]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:52:33.932 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:33 smithi028 ceph-mon[66336]: Health check failed: Degraded data redundancy: 37/1509 objects degraded (2.452%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:34.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 277324 9.82 MB/sec execute 568 sec latency 68.621 ms 2024-02-10T05:52:34.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:33 smithi192 ceph-mon[52881]: failed to encode map e146 with expected crc 2024-02-10T05:52:34.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:33 smithi192 ceph-mon[52881]: osdmap e146: 8 total, 7 up, 8 in 2024-02-10T05:52:34.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:33 smithi192 ceph-mon[52881]: failed to encode map e146 with expected crc 2024-02-10T05:52:34.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:33 smithi192 ceph-mon[52881]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:52:34.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:33 smithi192 ceph-mon[52881]: Health check failed: Degraded data redundancy: 37/1509 objects degraded (2.452%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:34.322 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:33 smithi192 podman[62154]: 2024-02-10 05:52:33.858200127 +0000 UTC m=+0.409928041 container init 86e48d87dc937ebfcd08597bf6546fd8fb70d056347d4091b841ddbc0aa4aaea (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-10T05:52:34.322 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:33 smithi192 podman[62154]: 2024-02-10 05:52:33.862724681 +0000 UTC m=+0.414452581 container start 86e48d87dc937ebfcd08597bf6546fd8fb70d056347d4091b841ddbc0aa4aaea (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:52:34.322 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:33 smithi192 podman[62154]: 2024-02-10 05:52:33.96657261 +0000 UTC m=+0.518300512 container attach 86e48d87dc937ebfcd08597bf6546fd8fb70d056347d4091b841ddbc0aa4aaea (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0) 2024-02-10T05:52:34.721 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate[62168]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T05:52:34.721 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate[62168]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T05:52:34.721 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 bash[62154]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T05:52:34.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 bash[62154]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-10T05:52:34.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 bash[62154]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T05:52:34.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 bash[62154]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T05:52:34.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 bash[62154]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-10T05:52:34.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 bash[62154]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T05:52:34.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 bash[62154]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-10T05:52:34.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate[62168]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-10T05:52:34.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate[62168]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-10T05:52:34.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate[62168]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-10T05:52:34.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate[62168]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-10T05:52:34.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate[62168]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-10T05:52:34.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:34 smithi192 podman[62154]: 2024-02-10 05:52:34.403714179 +0000 UTC m=+0.955442072 container died 86e48d87dc937ebfcd08597bf6546fd8fb70d056347d4091b841ddbc0aa4aaea (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-10T05:52:34.962 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:34 smithi028 ceph-mon[66336]: pgmap v257: 97 pgs: 3 active+undersized, 2 active+clean+remapped, 12 active+undersized+degraded, 9 peering, 5 stale+active+clean, 66 active+clean; 322 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 526 op/s; 95/1524 objects degraded (6.234%) 2024-02-10T05:52:35.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:34 smithi192 ceph-mon[52881]: pgmap v257: 97 pgs: 3 active+undersized, 2 active+clean+remapped, 12 active+undersized+degraded, 9 peering, 5 stale+active+clean, 66 active+clean; 322 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 526 op/s; 95/1524 objects degraded (6.234%) 2024-02-10T05:52:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 277624 9.82 MB/sec execute 569 sec latency 35.816 ms 2024-02-10T05:52:35.415 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:35 smithi192 podman[62291]: 2024-02-10 05:52:35.156231103 +0000 UTC m=+0.746162659 container remove 86e48d87dc937ebfcd08597bf6546fd8fb70d056347d4091b841ddbc0aa4aaea (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-activate, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:52:35.415 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:35 smithi192 podman[62356]: 2024-02-10 05:52:35.279246303 +0000 UTC m=+0.019700682 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:52:35.415 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:35 smithi192 podman[62356]: 2024-02-10 05:52:35.415641165 +0000 UTC m=+0.156095532 container create c9776c8b4f21fd88b8f9889f42108a2624ef3ca2fbd0b7016d94c750a2c4e2e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-10T05:52:35.680 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:35 smithi192 podman[62356]: 2024-02-10 05:52:35.651504801 +0000 UTC m=+0.391959170 container init c9776c8b4f21fd88b8f9889f42108a2624ef3ca2fbd0b7016d94c750a2c4e2e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6, org.label-schema.build-date=20240131, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-10T05:52:35.681 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:35 smithi192 podman[62356]: 2024-02-10 05:52:35.654924256 +0000 UTC m=+0.395378623 container start c9776c8b4f21fd88b8f9889f42108a2624ef3ca2fbd0b7016d94c750a2c4e2e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6, GIT_CLEAN=True, org.label-schema.build-date=20240131, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:52:36.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:35 smithi192 ceph-osd[62375]: -- 172.21.15.192:0/3405411185 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563092c911e0 con 0x563092c62c00 2024-02-10T05:52:36.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:35 smithi192 bash[62356]: c9776c8b4f21fd88b8f9889f42108a2624ef3ca2fbd0b7016d94c750a2c4e2e7 2024-02-10T05:52:36.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:35 smithi192 systemd[1]: Started Ceph osd.6 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:52:36.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 277914 9.81 MB/sec execute 570 sec latency 47.623 ms 2024-02-10T05:52:36.572 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:36 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:52:36.199+0000 7f4bced8b7c0 -1 Falling back to public interface 2024-02-10T05:52:36.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:36 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:36.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:36 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:36.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:36 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:36.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:36 smithi028 ceph-mon[66336]: pgmap v258: 97 pgs: 9 active+undersized, 9 active+clean+remapped, 24 active+undersized+degraded, 55 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 7.4 MiB/s wr, 752 op/s; 190/1698 objects degraded (11.190%); 3/1698 objects misplaced (0.177%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:52:37.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:36 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:36 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:36 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:37.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:36 smithi192 ceph-mon[52881]: pgmap v258: 97 pgs: 9 active+undersized, 9 active+clean+remapped, 24 active+undersized+degraded, 55 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 7.4 MiB/s wr, 752 op/s; 190/1698 objects degraded (11.190%); 3/1698 objects misplaced (0.177%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:52:37.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 278208 9.80 MB/sec execute 571 sec latency 44.234 ms 2024-02-10T05:52:37.886 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:37 smithi192 ceph-mon[52881]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T05:52:38.012 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:37 smithi028 ceph-mon[66336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T05:52:38.019 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:52:38.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 278461 9.79 MB/sec execute 572 sec latency 60.256 ms 2024-02-10T05:52:39.020 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:38 smithi028 ceph-mon[66336]: from='client.44339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:39.020 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:38 smithi028 ceph-mon[66336]: pgmap v259: 97 pgs: 9 active+undersized, 9 active+clean+remapped, 24 active+undersized+degraded, 55 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 4.9 MiB/s wr, 393 op/s; 190/1695 objects degraded (11.209%); 3/1695 objects misplaced (0.177%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:52:39.079 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:52:39.079 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (8m) 98s ago 18m 15.4M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:52:39.079 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (4m) 98s ago 17m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:52:39.079 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (4m) 37s ago 17m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:52:39.079 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (8m) 98s ago 17m 45.8M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:52:39.079 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (12m) 98s ago 12m 38.5M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:52:39.079 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (12m) 37s ago 12m 15.1M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:52:39.079 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (7m) 98s ago 19m 504M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (6m) 37s ago 17m 453M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (6m) 98s ago 19m 51.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (5m) 37s ago 17m 42.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (11m) 98s ago 11m 282M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (11m) 37s ago 11m 86.2M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (8m) 98s ago 17m 13.5M - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (7m) 37s ago 17m 21.4M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (4m) 98s ago 16m 339M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (3m) 98s ago 16m 289M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (2m) 98s ago 15m 233M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 a45f0e24ae44 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (108s) 98s ago 15m 11.6M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 25f4cf94969e 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (80s) 37s ago 14m 269M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 b15b2733ec77 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (42s) 37s ago 14m 12.6M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 000913de61d6 2024-02-10T05:52:39.080 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 starting - - - 2704M 2024-02-10T05:52:39.081 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (13m) 37s ago 13m 601M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:52:39.081 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (6m) 98s ago 16m 62.1M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:52:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 278788 9.79 MB/sec execute 573 sec latency 104.464 ms 2024-02-10T05:52:39.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:38 smithi192 ceph-mon[52881]: from='client.44339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:39.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:38 smithi192 ceph-mon[52881]: pgmap v259: 97 pgs: 9 active+undersized, 9 active+clean+remapped, 24 active+undersized+degraded, 55 active+clean; 335 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 4.9 MiB/s wr, 393 op/s; 190/1695 objects degraded (11.209%); 3/1695 objects misplaced (0.177%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:52:39.674 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:52:39.674 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:52:39.674 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:52:39.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:52:39.674 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:52:39.674 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:52:39.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:52:39.674 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:52:39.674 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T05:52:39.674 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 6 2024-02-10T05:52:39.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:52:39.675 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:52:39.675 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:52:39.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:52:39.675 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:52:39.675 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-10T05:52:39.675 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 10 2024-02-10T05:52:39.675 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:52:39.675 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:52:40.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:39 smithi028 ceph-mon[66336]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:40.033 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:39 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 190/1695 objects degraded (11.209%), 24 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:40.033 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:39 smithi028 ceph-mon[66336]: from='client.44345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:40.033 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:39 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/292956537' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:40.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 279263 9.78 MB/sec execute 574 sec latency 42.890 ms 2024-02-10T05:52:40.196 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:52:40.196 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:52:40.196 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:52:40.196 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:52:40.197 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:52:40.197 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:52:40.197 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:52:40.197 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:52:40.197 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:52:40.197 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "13/23 daemons upgraded", 2024-02-10T05:52:40.197 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T05:52:40.197 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:52:40.197 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:52:40.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:39 smithi192 ceph-mon[52881]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:40.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:39 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 190/1695 objects degraded (11.209%), 24 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:40.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:39 smithi192 ceph-mon[52881]: from='client.44345 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:40.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:39 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/292956537' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:40.904 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 192/1686 objects degraded (11.388%), 24 pgs degraded 2024-02-10T05:52:40.904 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-10T05:52:40.904 INFO:teuthology.orchestra.run.smithi028.stdout: osd.6 (root=default,host=smithi192) is down 2024-02-10T05:52:40.904 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 192/1686 objects degraded (11.388%), 24 pgs degraded 2024-02-10T05:52:40.904 INFO:teuthology.orchestra.run.smithi028.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-02-10T05:52:40.904 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-02-10T05:52:40.904 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-02-10T05:52:40.904 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.7 is active+undersized+degraded, acting [7,2] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.19 is active+undersized+degraded, acting [3,0] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1b is active+undersized+degraded, acting [3,7] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-02-10T05:52:40.905 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-02-10T05:52:40.906 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-02-10T05:52:40.906 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-02-10T05:52:40.906 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-02-10T05:52:40.906 INFO:teuthology.orchestra.run.smithi028.stdout: pg 4.5 is active+undersized+degraded, acting [4,7] 2024-02-10T05:52:41.183 INFO:teuthology.orchestra.run.smithi028.stdout: 5 279632 9.78 MB/sec execute 575 sec latency 44.938 ms 2024-02-10T05:52:41.305 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:41 smithi028 ceph-mon[66336]: pgmap v260: 97 pgs: 9 active+undersized, 9 active+clean+remapped, 24 active+undersized+degraded, 55 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.4 MiB/s wr, 362 op/s; 192/1686 objects degraded (11.388%); 3/1686 objects misplaced (0.178%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:52:41.305 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:41 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:41.305 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:41 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:41.306 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:41 smithi028 ceph-mon[66336]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:41.306 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:41 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2191795417' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:52:41.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:41 smithi192 ceph-mon[52881]: pgmap v260: 97 pgs: 9 active+undersized, 9 active+clean+remapped, 24 active+undersized+degraded, 55 active+clean; 334 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 4.4 MiB/s wr, 362 op/s; 192/1686 objects degraded (11.388%); 3/1686 objects misplaced (0.178%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:52:41.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:41 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:41.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:41 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:41.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:41 smithi192 ceph-mon[52881]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:41.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:41 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2191795417' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:52:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 280001 9.78 MB/sec execute 576 sec latency 33.072 ms 2024-02-10T05:52:42.519 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:42 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:42.519 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:42 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:42.519 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:42 smithi192 ceph-mon[52881]: pgmap v261: 97 pgs: 9 active+undersized, 9 active+clean+remapped, 24 active+undersized+degraded, 55 active+clean; 330 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.8 MiB/s wr, 445 op/s; 201/1605 objects degraded (12.523%); 3/1605 objects misplaced (0.187%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:52:42.589 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:42 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:42.589 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:42 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:42.589 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:42 smithi028 ceph-mon[66336]: pgmap v261: 97 pgs: 9 active+undersized, 9 active+clean+remapped, 24 active+undersized+degraded, 55 active+clean; 330 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.8 MiB/s wr, 445 op/s; 201/1605 objects degraded (12.523%); 3/1605 objects misplaced (0.187%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:52:43.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 280370 9.79 MB/sec execute 577 sec latency 35.392 ms 2024-02-10T05:52:44.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 280733 9.80 MB/sec execute 578 sec latency 63.159 ms 2024-02-10T05:52:44.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:44 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 202/1611 objects degraded (12.539%), 24 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:44.363 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:44 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 202/1611 objects degraded (12.539%), 24 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:44.984 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:44 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:52:44.798+0000 7f4bced8b7c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-02-10T05:52:45.141 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:45 smithi028 ceph-mon[66336]: pgmap v262: 97 pgs: 9 active+undersized, 9 active+clean+remapped, 24 active+undersized+degraded, 55 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 4.7 MiB/s wr, 436 op/s; 202/1611 objects degraded (12.539%); 3/1611 objects misplaced (0.186%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:52:45.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:45.142 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:52:45.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 280999 9.81 MB/sec execute 579 sec latency 54.034 ms 2024-02-10T05:52:45.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:45 smithi192 ceph-mon[52881]: pgmap v262: 97 pgs: 9 active+undersized, 9 active+clean+remapped, 24 active+undersized+degraded, 55 active+clean; 325 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 4.7 MiB/s wr, 436 op/s; 202/1611 objects degraded (12.539%); 3/1611 objects misplaced (0.186%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:52:45.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:45.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:52:46.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 281245 9.80 MB/sec execute 580 sec latency 37.657 ms 2024-02-10T05:52:46.322 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:45 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:52:45.973+0000 7f4bced8b7c0 -1 osd.6 142 log_to_monitors true 2024-02-10T05:52:46.415 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:46.415 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:46.415 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:46.415 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:46.415 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:46.415 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:46.415 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:46.415 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:46.415 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:46.415 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:46.416 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T05:52:46.416 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T05:52:46.416 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T05:52:46.416 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='osd.6 [v2:172.21.15.192:6816/2822684876,v1:172.21.15.192:6817/2822684876]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:52:46.416 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:52:46.416 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:46 smithi028 ceph-mon[66336]: pgmap v263: 97 pgs: 9 active+undersized, 9 active+clean+remapped, 24 active+undersized+degraded, 55 active+clean; 334 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 6.1 MiB/s wr, 544 op/s; 201/1692 objects degraded (11.879%); 3/1692 objects misplaced (0.177%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:52:46.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:46.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:46.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:46.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:46.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:46.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:52:46.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:46.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:46.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:46.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:52:46.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T05:52:46.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T05:52:46.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-10T05:52:46.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='osd.6 [v2:172.21.15.192:6816/2822684876,v1:172.21.15.192:6817/2822684876]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:52:46.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:52:46.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-mon[52881]: pgmap v263: 97 pgs: 9 active+undersized, 9 active+clean+remapped, 24 active+undersized+degraded, 55 active+clean; 334 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 6.1 MiB/s wr, 544 op/s; 201/1692 objects degraded (11.879%); 3/1692 objects misplaced (0.177%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-10T05:52:47.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 281427 9.79 MB/sec execute 581 sec latency 157.418 ms 2024-02-10T05:52:47.322 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:52:46 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:52:46.922+0000 7f4bbc1db700 -1 osd.6 142 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:52:47.449 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:47 smithi028 ceph-mon[66336]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:52:47.450 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:47 smithi028 ceph-mon[66336]: failed to encode map e147 with expected crc 2024-02-10T05:52:47.450 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:47 smithi028 ceph-mon[66336]: osdmap e147: 8 total, 7 up, 8 in 2024-02-10T05:52:47.450 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:47 smithi028 ceph-mon[66336]: from='osd.6 [v2:172.21.15.192:6816/2822684876,v1:172.21.15.192:6817/2822684876]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:52:47.450 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:47 smithi028 ceph-mon[66336]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:52:47.450 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:47 smithi028 ceph-mon[66336]: from='osd.6 ' entity='osd.6' 2024-02-10T05:52:47.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:47 smithi192 ceph-mon[52881]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:52:47.824 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:47 smithi192 ceph-mon[52881]: failed to encode map e147 with expected crc 2024-02-10T05:52:47.824 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:47 smithi192 ceph-mon[52881]: osdmap e147: 8 total, 7 up, 8 in 2024-02-10T05:52:47.824 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:47 smithi192 ceph-mon[52881]: from='osd.6 [v2:172.21.15.192:6816/2822684876,v1:172.21.15.192:6817/2822684876]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:52:47.824 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:47 smithi192 ceph-mon[52881]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:52:47.824 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:47 smithi192 ceph-mon[52881]: from='osd.6 ' entity='osd.6' 2024-02-10T05:52:48.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 281651 9.77 MB/sec execute 582 sec latency 232.695 ms 2024-02-10T05:52:49.133 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:48 smithi028 ceph-mon[66336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:52:49.134 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:48 smithi028 ceph-mon[66336]: osd.6 [v2:172.21.15.192:6816/2822684876,v1:172.21.15.192:6817/2822684876] boot 2024-02-10T05:52:49.134 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:48 smithi028 ceph-mon[66336]: osdmap e148: 8 total, 8 up, 8 in 2024-02-10T05:52:49.134 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:52:49.134 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:48 smithi028 ceph-mon[66336]: failed to encode map e148 with expected crc 2024-02-10T05:52:49.134 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:48 smithi028 ceph-mon[66336]: failed to encode map e148 with expected crc 2024-02-10T05:52:49.134 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:48 smithi028 ceph-mon[66336]: pgmap v266: 97 pgs: 1 remapped+peering, 5 peering, 8 active+undersized, 8 active+clean+remapped, 20 active+undersized+degraded, 55 active+clean; 335 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 5.3 MiB/s wr, 479 op/s; 163/1701 objects degraded (9.583%); 3/1701 objects misplaced (0.176%) 2024-02-10T05:52:49.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 281683 9.76 MB/sec execute 583 sec latency 1133.317 ms 2024-02-10T05:52:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:48 smithi192 ceph-mon[52881]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:52:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:48 smithi192 ceph-mon[52881]: osd.6 [v2:172.21.15.192:6816/2822684876,v1:172.21.15.192:6817/2822684876] boot 2024-02-10T05:52:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:48 smithi192 ceph-mon[52881]: osdmap e148: 8 total, 8 up, 8 in 2024-02-10T05:52:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:52:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:48 smithi192 ceph-mon[52881]: failed to encode map e148 with expected crc 2024-02-10T05:52:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:48 smithi192 ceph-mon[52881]: failed to encode map e148 with expected crc 2024-02-10T05:52:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:48 smithi192 ceph-mon[52881]: pgmap v266: 97 pgs: 1 remapped+peering, 5 peering, 8 active+undersized, 8 active+clean+remapped, 20 active+undersized+degraded, 55 active+clean; 335 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 5.3 MiB/s wr, 479 op/s; 163/1701 objects degraded (9.583%); 3/1701 objects misplaced (0.176%) 2024-02-10T05:52:50.182 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:49 smithi028 ceph-mon[66336]: failed to encode map e149 with expected crc 2024-02-10T05:52:50.182 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:49 smithi028 ceph-mon[66336]: osdmap e149: 8 total, 8 up, 8 in 2024-02-10T05:52:50.183 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:49 smithi028 ceph-mon[66336]: failed to encode map e149 with expected crc 2024-02-10T05:52:50.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 281816 9.75 MB/sec execute 584 sec latency 1939.208 ms 2024-02-10T05:52:50.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:49 smithi192 ceph-mon[52881]: failed to encode map e149 with expected crc 2024-02-10T05:52:50.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:49 smithi192 ceph-mon[52881]: osdmap e149: 8 total, 8 up, 8 in 2024-02-10T05:52:50.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:49 smithi192 ceph-mon[52881]: failed to encode map e149 with expected crc 2024-02-10T05:52:50.968 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:50 smithi028 ceph-mon[66336]: failed to encode map e150 with expected crc 2024-02-10T05:52:50.969 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:50 smithi028 ceph-mon[66336]: osdmap e150: 8 total, 8 up, 8 in 2024-02-10T05:52:50.969 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:50 smithi028 ceph-mon[66336]: failed to encode map e150 with expected crc 2024-02-10T05:52:50.969 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:50 smithi028 ceph-mon[66336]: failed to encode map e150 with expected crc 2024-02-10T05:52:50.969 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:50 smithi028 ceph-mon[66336]: failed to encode map e150 with expected crc 2024-02-10T05:52:50.969 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:50 smithi028 ceph-mon[66336]: failed to encode map e150 with expected crc 2024-02-10T05:52:50.969 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:50 smithi028 ceph-mon[66336]: pgmap v269: 97 pgs: 2 remapped+peering, 8 peering, 3 active+undersized, 7 active+clean+remapped, 16 active+undersized+degraded, 61 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s wr, 267 op/s; 118/1698 objects degraded (6.949%); 3/1698 objects misplaced (0.177%); 154 KiB/s, 7 objects/s recovering 2024-02-10T05:52:51.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 282071 9.74 MB/sec execute 585 sec latency 58.340 ms 2024-02-10T05:52:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:50 smithi192 ceph-mon[52881]: failed to encode map e150 with expected crc 2024-02-10T05:52:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:50 smithi192 ceph-mon[52881]: osdmap e150: 8 total, 8 up, 8 in 2024-02-10T05:52:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:50 smithi192 ceph-mon[52881]: failed to encode map e150 with expected crc 2024-02-10T05:52:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:50 smithi192 ceph-mon[52881]: failed to encode map e150 with expected crc 2024-02-10T05:52:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:50 smithi192 ceph-mon[52881]: failed to encode map e150 with expected crc 2024-02-10T05:52:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:50 smithi192 ceph-mon[52881]: failed to encode map e150 with expected crc 2024-02-10T05:52:51.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:50 smithi192 ceph-mon[52881]: pgmap v269: 97 pgs: 2 remapped+peering, 8 peering, 3 active+undersized, 7 active+clean+remapped, 16 active+undersized+degraded, 61 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s wr, 267 op/s; 118/1698 objects degraded (6.949%); 3/1698 objects misplaced (0.177%); 154 KiB/s, 7 objects/s recovering 2024-02-10T05:52:51.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:51 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 118/1698 objects degraded (6.949%), 16 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:51.991 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:51 smithi028 ceph-mon[66336]: osdmap e151: 8 total, 8 up, 8 in 2024-02-10T05:52:51.991 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:51 smithi028 ceph-mon[66336]: failed to encode map e151 with expected crc 2024-02-10T05:52:51.991 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:51 smithi028 ceph-mon[66336]: failed to encode map e151 with expected crc 2024-02-10T05:52:51.991 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:51 smithi028 ceph-mon[66336]: failed to encode map e151 with expected crc 2024-02-10T05:52:52.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 282428 9.74 MB/sec execute 586 sec latency 771.532 ms 2024-02-10T05:52:52.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:51 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 118/1698 objects degraded (6.949%), 16 pgs degraded (PG_DEGRADED) 2024-02-10T05:52:52.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:51 smithi192 ceph-mon[52881]: osdmap e151: 8 total, 8 up, 8 in 2024-02-10T05:52:52.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:51 smithi192 ceph-mon[52881]: failed to encode map e151 with expected crc 2024-02-10T05:52:52.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:51 smithi192 ceph-mon[52881]: failed to encode map e151 with expected crc 2024-02-10T05:52:52.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:51 smithi192 ceph-mon[52881]: failed to encode map e151 with expected crc 2024-02-10T05:52:53.002 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:52 smithi028 ceph-mon[66336]: failed to encode map e152 with expected crc 2024-02-10T05:52:53.003 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:52 smithi028 ceph-mon[66336]: osdmap e152: 8 total, 8 up, 8 in 2024-02-10T05:52:53.003 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:52 smithi028 ceph-mon[66336]: failed to encode map e152 with expected crc 2024-02-10T05:52:53.003 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:52 smithi028 ceph-mon[66336]: pgmap v272: 97 pgs: 11 peering, 3 active+undersized+remapped, 83 active+clean; 330 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 3.9 MiB/s wr, 543 op/s; 2.0 MiB/s, 34 objects/s recovering 2024-02-10T05:52:53.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 282762 9.73 MB/sec execute 587 sec latency 54.269 ms 2024-02-10T05:52:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:52 smithi192 ceph-mon[52881]: failed to encode map e152 with expected crc 2024-02-10T05:52:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:52 smithi192 ceph-mon[52881]: osdmap e152: 8 total, 8 up, 8 in 2024-02-10T05:52:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:52 smithi192 ceph-mon[52881]: failed to encode map e152 with expected crc 2024-02-10T05:52:53.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:52 smithi192 ceph-mon[52881]: pgmap v272: 97 pgs: 11 peering, 3 active+undersized+remapped, 83 active+clean; 330 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 3.9 MiB/s wr, 543 op/s; 2.0 MiB/s, 34 objects/s recovering 2024-02-10T05:52:54.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:53 smithi028 ceph-mon[66336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 118/1698 objects degraded (6.949%), 16 pgs degraded) 2024-02-10T05:52:54.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:53 smithi028 ceph-mon[66336]: Cluster is now healthy 2024-02-10T05:52:54.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 283199 9.73 MB/sec execute 588 sec latency 52.821 ms 2024-02-10T05:52:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:53 smithi192 ceph-mon[52881]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 118/1698 objects degraded (6.949%), 16 pgs degraded) 2024-02-10T05:52:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:53 smithi192 ceph-mon[52881]: Cluster is now healthy 2024-02-10T05:52:55.120 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:54 smithi028 ceph-mon[66336]: pgmap v273: 97 pgs: 3 peering, 3 active+undersized+remapped, 91 active+clean; 328 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 2.3 MiB/s wr, 271 op/s; 1.9 MiB/s, 59 keys/s, 39 objects/s recovering 2024-02-10T05:52:55.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 283501 9.73 MB/sec execute 589 sec latency 46.449 ms 2024-02-10T05:52:55.222 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:54 smithi192 ceph-mon[52881]: pgmap v273: 97 pgs: 3 peering, 3 active+undersized+remapped, 91 active+clean; 328 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 2.3 MiB/s wr, 271 op/s; 1.9 MiB/s, 59 keys/s, 39 objects/s recovering 2024-02-10T05:52:56.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 283851 9.74 MB/sec execute 590 sec latency 51.781 ms 2024-02-10T05:52:57.160 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:57 smithi028 ceph-mon[66336]: pgmap v274: 97 pgs: 1 peering, 96 active+clean; 324 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.0 MiB/s wr, 367 op/s; 1.5 MiB/s, 49 keys/s, 28 objects/s recovering 2024-02-10T05:52:57.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 284199 9.74 MB/sec execute 591 sec latency 51.916 ms 2024-02-10T05:52:57.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:57 smithi192 ceph-mon[52881]: pgmap v274: 97 pgs: 1 peering, 96 active+clean; 324 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 4.0 MiB/s wr, 367 op/s; 1.5 MiB/s, 49 keys/s, 28 objects/s recovering 2024-02-10T05:52:58.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 284546 9.75 MB/sec execute 592 sec latency 73.320 ms 2024-02-10T05:52:58.955 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:52:58 smithi028 ceph-mon[66336]: pgmap v275: 97 pgs: 97 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.0 MiB/s wr, 490 op/s; 1.1 MiB/s, 37 keys/s, 23 objects/s recovering 2024-02-10T05:52:59.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 284829 9.75 MB/sec execute 593 sec latency 38.918 ms 2024-02-10T05:52:59.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:52:58 smithi192 ceph-mon[52881]: pgmap v275: 97 pgs: 97 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.0 MiB/s wr, 490 op/s; 1.1 MiB/s, 37 keys/s, 23 objects/s recovering 2024-02-10T05:53:00.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 285067 9.74 MB/sec execute 594 sec latency 41.912 ms 2024-02-10T05:53:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 285322 9.73 MB/sec execute 595 sec latency 54.682 ms 2024-02-10T05:53:01.230 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:01 smithi028 ceph-mon[66336]: pgmap v276: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.3 MiB/s wr, 354 op/s; 662 KiB/s, 33 keys/s, 16 objects/s recovering 2024-02-10T05:53:01.230 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:01.231 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:53:01.231 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T05:53:01.311 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:01 smithi192 ceph-mon[52881]: pgmap v276: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.3 MiB/s wr, 354 op/s; 662 KiB/s, 33 keys/s, 16 objects/s recovering 2024-02-10T05:53:01.311 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:01.311 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:53:01.311 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T05:53:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 285597 9.72 MB/sec execute 596 sec latency 53.055 ms 2024-02-10T05:53:02.254 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:02 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T05:53:02.254 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:02 smithi028 ceph-mon[66336]: Upgrade: osd.7 is safe to restart 2024-02-10T05:53:02.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:02 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-10T05:53:02.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:02 smithi192 ceph-mon[52881]: Upgrade: osd.7 is safe to restart 2024-02-10T05:53:03.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 285965 9.72 MB/sec execute 597 sec latency 48.695 ms 2024-02-10T05:53:03.272 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:03 smithi028 ceph-mon[66336]: pgmap v277: 97 pgs: 97 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.7 MiB/s wr, 454 op/s; 155 KiB/s, 30 keys/s, 7 objects/s recovering 2024-02-10T05:53:03.272 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:03 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:03.273 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:03 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:53:03.273 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:03 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:03.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:03 smithi192 ceph-mon[52881]: pgmap v277: 97 pgs: 97 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.7 MiB/s wr, 454 op/s; 155 KiB/s, 30 keys/s, 7 objects/s recovering 2024-02-10T05:53:03.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:03 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:03.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:03 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:53:03.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:03 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:04.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 286292 9.71 MB/sec execute 598 sec latency 60.473 ms 2024-02-10T05:53:04.288 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:04 smithi028 ceph-mon[66336]: Upgrade: Updating osd.7 2024-02-10T05:53:04.288 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:04 smithi028 ceph-mon[66336]: Deploying daemon osd.7 on smithi192 2024-02-10T05:53:04.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:04 smithi192 ceph-mon[52881]: Upgrade: Updating osd.7 2024-02-10T05:53:04.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:04 smithi192 ceph-mon[52881]: Deploying daemon osd.7 on smithi192 2024-02-10T05:53:05.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 286797 9.72 MB/sec execute 599 sec latency 39.314 ms 2024-02-10T05:53:05.253 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:05 smithi192 ceph-mon[52881]: pgmap v278: 97 pgs: 97 active+clean; 336 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.2 MiB/s wr, 398 op/s; 130 KiB/s, 25 keys/s, 6 objects/s recovering 2024-02-10T05:53:05.320 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:05 smithi028 ceph-mon[66336]: pgmap v278: 97 pgs: 97 active+clean; 336 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.2 MiB/s wr, 398 op/s; 130 KiB/s, 25 keys/s, 6 objects/s recovering 2024-02-10T05:53:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: 5 cleanup 600 sec 2024-02-10T05:53:06.268 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:06 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:06.334 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:06 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: 0 cleanup 600 sec 2024-02-10T05:53:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:53:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: Operation Count AvgLat MaxLat 2024-02-10T05:53:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: ---------------------------------------- 2024-02-10T05:53:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: NTCreateX 187343 4.563 2316.954 2024-02-10T05:53:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: Close 137597 0.141 10.637 2024-02-10T05:53:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: Rename 7933 10.153 1117.096 2024-02-10T05:53:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: Unlink 37837 6.814 1104.535 2024-02-10T05:53:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: Qpathinfo 169888 1.172 2315.111 2024-02-10T05:53:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: Qfileinfo 29605 0.008 2.308 2024-02-10T05:53:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: Qfsinfo 31113 1.349 4465.149 2024-02-10T05:53:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: Sfileinfo 15273 13.196 1044.965 2024-02-10T05:53:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: Find 65664 1.017 19.063 2024-02-10T05:53:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: WriteX 92541 13.733 2770.201 2024-02-10T05:53:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: ReadX 293415 0.010 6.048 2024-02-10T05:53:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: LockX 608 0.399 2.013 2024-02-10T05:53:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: UnlockX 608 0.349 3.139 2024-02-10T05:53:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: Flush 13139 0.002 0.056 2024-02-10T05:53:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:53:06.431 INFO:teuthology.orchestra.run.smithi028.stdout:Throughput 9.7162 MB/sec 5 clients 5 procs max_latency=4465.153 ms 2024-02-10T05:53:06.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-02-10T05:53:06.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-02-10T05:53:06.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-02-10T05:53:06.618 INFO:teuthology.orchestra.run.smithi028.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-02-10T05:53:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:53:06.619 INFO:teuthology.orchestra.run.smithi028.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-02-10T05:53:06.619 INFO:teuthology.orchestra.run.smithi028.stdout:3 of 5 processes prepared for launch 0 sec 2024-02-10T05:53:06.719 INFO:teuthology.orchestra.run.smithi028.stdout:5 of 5 processes prepared for launch 0 sec 2024-02-10T05:53:06.720 INFO:teuthology.orchestra.run.smithi028.stdout:releasing clients 2024-02-10T05:53:06.900 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:06 smithi192 systemd[1]: Stopping Ceph osd.7 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:53:07.150 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:06 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[40120]: 2024-02-10T05:53:06.898+0000 7f27cf1a1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:53:07.151 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:06 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[40120]: 2024-02-10T05:53:06.898+0000 7f27cf1a1700 -1 osd.7 152 *** Got signal Terminated *** 2024-02-10T05:53:07.151 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:06 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[40120]: 2024-02-10T05:53:06.898+0000 7f27cf1a1700 -1 osd.7 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:53:07.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:07 smithi028 ceph-mon[66336]: pgmap v279: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 5.2 MiB/s wr, 479 op/s; 682 B/s, 1 objects/s recovering 2024-02-10T05:53:07.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:07 smithi028 ceph-mon[66336]: osd.7 marked itself down and dead 2024-02-10T05:53:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:07 smithi192 ceph-mon[52881]: pgmap v279: 97 pgs: 97 active+clean; 331 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 5.2 MiB/s wr, 479 op/s; 682 B/s, 1 objects/s recovering 2024-02-10T05:53:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:07 smithi192 ceph-mon[52881]: osd.7 marked itself down and dead 2024-02-10T05:53:07.999 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:07 smithi192 podman[64192]: 2024-02-10 05:53:07.712378882 +0000 UTC m=+0.933241272 container died 0d58f3582c2d07804358e06a64d5ee5afea0ced28280dfdbb30cef684b88e5cd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, release=754, name=centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, io.openshift.expose-services=, io.buildah.version=1.19.8, ceph=True, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, version=8, build-date=2022-05-03T08:36:31.336870) 2024-02-10T05:53:08.301 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:08 smithi028 ceph-mon[66336]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:53:08.301 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:08 smithi028 ceph-mon[66336]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-10T05:53:08.302 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:08 smithi028 ceph-mon[66336]: osdmap e153: 8 total, 7 up, 8 in 2024-02-10T05:53:08.302 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:08 smithi028 ceph-mon[66336]: osdmap e154: 8 total, 7 up, 8 in 2024-02-10T05:53:08.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:08 smithi192 ceph-mon[52881]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T05:53:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:08 smithi192 ceph-mon[52881]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-10T05:53:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:08 smithi192 ceph-mon[52881]: osdmap e153: 8 total, 7 up, 8 in 2024-02-10T05:53:08.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:08 smithi192 ceph-mon[52881]: osdmap e154: 8 total, 7 up, 8 in 2024-02-10T05:53:08.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:08 smithi192 podman[64192]: 2024-02-10 05:53:08.090259472 +0000 UTC m=+1.311121860 container cleanup 0d58f3582c2d07804358e06a64d5ee5afea0ced28280dfdbb30cef684b88e5cd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, name=centos-stream, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, release=754, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc.) 2024-02-10T05:53:08.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:08 smithi192 bash[64192]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7 2024-02-10T05:53:08.714 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:08 smithi192 podman[64209]: 2024-02-10 05:53:08.384171474 +0000 UTC m=+0.667369087 container remove 0d58f3582c2d07804358e06a64d5ee5afea0ced28280dfdbb30cef684b88e5cd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.openshift.expose-services=, release=754, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, vcs-type=git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, architecture=x86_64) 2024-02-10T05:53:08.719 INFO:teuthology.orchestra.run.smithi028.stdout: 5 231 13.00 MB/sec execute 1 sec latency 1047.028 ms 2024-02-10T05:53:09.005 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:08 smithi192 podman[64302]: 2024-02-10 05:53:08.619383051 +0000 UTC m=+0.020822499 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:53:09.005 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:08 smithi192 podman[64302]: 2024-02-10 05:53:08.752137848 +0000 UTC m=+0.153577336 container create 9544f66a0f1020c8dc1582d1169a91a36fa034599f233ff7329e6085a2a997f9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD) 2024-02-10T05:53:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:09 smithi192 ceph-mon[52881]: pgmap v281: 97 pgs: 8 peering, 11 stale+active+clean, 78 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 MiB/s wr, 456 op/s 2024-02-10T05:53:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:09 smithi192 ceph-mon[52881]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:53:09.323 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:09 smithi192 podman[64302]: 2024-02-10 05:53:09.005322843 +0000 UTC m=+0.406762289 container init 9544f66a0f1020c8dc1582d1169a91a36fa034599f233ff7329e6085a2a997f9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-10T05:53:09.323 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:09 smithi192 podman[64302]: 2024-02-10 05:53:09.009852953 +0000 UTC m=+0.411292411 container start 9544f66a0f1020c8dc1582d1169a91a36fa034599f233ff7329e6085a2a997f9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a) 2024-02-10T05:53:09.323 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:09 smithi192 podman[64302]: 2024-02-10 05:53:09.138899073 +0000 UTC m=+0.540338519 container attach 9544f66a0f1020c8dc1582d1169a91a36fa034599f233ff7329e6085a2a997f9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-10T05:53:09.393 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:09 smithi028 ceph-mon[66336]: pgmap v281: 97 pgs: 8 peering, 11 stale+active+clean, 78 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 MiB/s wr, 456 op/s 2024-02-10T05:53:09.393 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:09 smithi028 ceph-mon[66336]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:53:09.720 INFO:teuthology.orchestra.run.smithi028.stdout: 5 435 22.63 MB/sec execute 2 sec latency 61.060 ms 2024-02-10T05:53:09.751 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:09 smithi192 podman[64302]: 2024-02-10 05:53:09.379551813 +0000 UTC m=+0.780991261 container died 9544f66a0f1020c8dc1582d1169a91a36fa034599f233ff7329e6085a2a997f9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-10T05:53:10.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:10 smithi192 ceph-mon[52881]: Health check failed: Degraded data redundancy: 42/1428 objects degraded (2.941%), 11 pgs degraded (PG_DEGRADED) 2024-02-10T05:53:10.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:10 smithi192 podman[64327]: 2024-02-10 05:53:10.044523518 +0000 UTC m=+0.658934873 container remove 9544f66a0f1020c8dc1582d1169a91a36fa034599f233ff7329e6085a2a997f9 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0) 2024-02-10T05:53:10.323 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:10 smithi192 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.7.service: Deactivated successfully. 2024-02-10T05:53:10.323 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:10 smithi192 systemd[1]: Stopped Ceph osd.7 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:53:10.323 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:10 smithi192 systemd[1]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.7.service: Consumed 7min 51.519s CPU time. 2024-02-10T05:53:10.323 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:10 smithi192 systemd[1]: Starting Ceph osd.7 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:53:10.419 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:10 smithi028 ceph-mon[66336]: Health check failed: Degraded data redundancy: 42/1428 objects degraded (2.941%), 11 pgs degraded (PG_DEGRADED) 2024-02-10T05:53:10.720 INFO:teuthology.orchestra.run.smithi028.stdout: 5 744 25.07 MB/sec execute 3 sec latency 74.170 ms 2024-02-10T05:53:10.782 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:10 smithi192 podman[64429]: 2024-02-10 05:53:10.419481973 +0000 UTC m=+0.019535134 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:53:10.782 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:10 smithi192 podman[64429]: 2024-02-10 05:53:10.537236217 +0000 UTC m=+0.137289355 container create bba03c138bcbe13e8b0967e707749d77a0d804ecefbfb8ad7fb613a1151e1f33 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-10T05:53:10.782 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:10 smithi192 podman[64429]: 2024-02-10 05:53:10.782183063 +0000 UTC m=+0.382236223 container init bba03c138bcbe13e8b0967e707749d77a0d804ecefbfb8ad7fb613a1151e1f33 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-10T05:53:11.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:10 smithi192 podman[64429]: 2024-02-10 05:53:10.786725036 +0000 UTC m=+0.386778180 container start bba03c138bcbe13e8b0967e707749d77a0d804ecefbfb8ad7fb613a1151e1f33 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:53:11.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:10 smithi192 podman[64429]: 2024-02-10 05:53:10.890823479 +0000 UTC m=+0.490876624 container attach bba03c138bcbe13e8b0967e707749d77a0d804ecefbfb8ad7fb613a1151e1f33 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.build-date=20240131) 2024-02-10T05:53:11.424 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:11 smithi028 ceph-mon[66336]: pgmap v283: 97 pgs: 11 active+undersized+degraded, 8 peering, 5 stale+active+clean, 73 active+clean; 319 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 5.4 MiB/s wr, 376 op/s; 42/1428 objects degraded (2.941%) 2024-02-10T05:53:11.518 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:53:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:11 smithi192 ceph-mon[52881]: pgmap v283: 97 pgs: 11 active+undersized+degraded, 8 peering, 5 stale+active+clean, 73 active+clean; 319 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 5.4 MiB/s wr, 376 op/s; 42/1428 objects degraded (2.941%) 2024-02-10T05:53:11.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate[64444]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T05:53:11.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate[64444]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T05:53:11.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 bash[64429]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T05:53:11.573 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 bash[64429]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T05:53:11.573 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 bash[64429]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T05:53:11.573 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 bash[64429]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T05:53:11.573 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 bash[64429]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-10T05:53:11.573 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 bash[64429]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T05:53:11.573 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 bash[64429]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-10T05:53:11.573 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate[64444]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T05:53:11.573 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate[64444]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T05:53:11.573 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate[64444]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-10T05:53:11.573 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate[64444]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-10T05:53:11.573 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate[64444]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-10T05:53:11.573 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 podman[64429]: 2024-02-10 05:53:11.345918435 +0000 UTC m=+0.945971567 container died bba03c138bcbe13e8b0967e707749d77a0d804ecefbfb8ad7fb613a1151e1f33 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-10T05:53:11.720 INFO:teuthology.orchestra.run.smithi028.stdout: 5 1312 22.47 MB/sec execute 4 sec latency 20.476 ms 2024-02-10T05:53:12.264 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:11 smithi192 podman[64576]: 2024-02-10 05:53:11.996296535 +0000 UTC m=+0.645119235 container remove bba03c138bcbe13e8b0967e707749d77a0d804ecefbfb8ad7fb613a1151e1f33 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, RELEASE=main-290212a) 2024-02-10T05:53:12.264 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:12 smithi192 podman[64635]: 2024-02-10 05:53:12.132293468 +0000 UTC m=+0.021735205 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:53:12.264 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:12 smithi192 podman[64635]: 2024-02-10 05:53:12.264253186 +0000 UTC m=+0.153694916 container create 86dbef76e22dfa12abbc808c06ea14f26ad230c0af97146fa4b5493867dd97e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, ceph=True) 2024-02-10T05:53:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:53:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (9m) 2m ago 18m 15.4M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:53:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (5m) 2m ago 18m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:53:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (5m) 32s ago 17m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:53:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (8m) 2m ago 18m 45.8M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:53:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (12m) 2m ago 12m 38.5M - 17.2.0 e1d6a67b021e d73188a9d69f 2024-02-10T05:53:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (12m) 32s ago 12m 15.3M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:53:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (8m) 2m ago 19m 504M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:53:12.491 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (7m) 32s ago 17m 454M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (7m) 2m ago 19m 51.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (6m) 32s ago 17m 41.8M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (12m) 2m ago 12m 282M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (12m) 32s ago 12m 86.3M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (8m) 2m ago 18m 13.5M - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (8m) 32s ago 17m 17.9M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (4m) 2m ago 17m 339M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (3m) 2m ago 16m 289M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (3m) 2m ago 16m 233M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 a45f0e24ae44 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (2m) 2m ago 15m 11.6M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 25f4cf94969e 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (113s) 32s ago 15m 290M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 b15b2733ec77 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (75s) 32s ago 14m 266M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 000913de61d6 2024-02-10T05:53:12.492 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (37s) 32s ago 14m 12.2M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 c9776c8b4f21 2024-02-10T05:53:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (14m) 32s ago 14m 653M 2704M 17.2.0 e1d6a67b021e 0d58f3582c2d 2024-02-10T05:53:12.493 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (7m) 2m ago 17m 62.1M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:53:12.720 INFO:teuthology.orchestra.run.smithi028.stdout: 5 cleanup 5 sec 2024-02-10T05:53:12.822 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:12 smithi192 podman[64635]: 2024-02-10 05:53:12.517543511 +0000 UTC m=+0.406985254 container init 86dbef76e22dfa12abbc808c06ea14f26ad230c0af97146fa4b5493867dd97e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-290212a) 2024-02-10T05:53:12.822 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:12 smithi192 podman[64635]: 2024-02-10 05:53:12.520908217 +0000 UTC m=+0.410349946 container start 86dbef76e22dfa12abbc808c06ea14f26ad230c0af97146fa4b5493867dd97e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T05:53:12.822 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:12 smithi192 ceph-osd[64654]: -- 172.21.15.192:0/2657462729 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5580f1db91e0 con 0x5580f1d8ac00 2024-02-10T05:53:12.822 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:12 smithi192 bash[64635]: 86dbef76e22dfa12abbc808c06ea14f26ad230c0af97146fa4b5493867dd97e4 2024-02-10T05:53:12.822 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:12 smithi192 systemd[1]: Started Ceph osd.7 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715. 2024-02-10T05:53:12.906 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c74900000035' failed - Device or resource busy 2024-02-10T05:53:12.907 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c74500000028' failed - Device or resource busy 2024-02-10T05:53:12.907 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c72f00000022' failed - Device or resource busy 2024-02-10T05:53:12.909 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c72400000048' failed - Device or resource busy 2024-02-10T05:53:12.909 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c77700000030' failed - Device or resource busy 2024-02-10T05:53:12.909 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c7620000002c' failed - Device or resource busy 2024-02-10T05:53:12.909 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c7560000001a' failed - Device or resource busy 2024-02-10T05:53:12.910 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c77200000026' failed - Device or resource busy 2024-02-10T05:53:12.910 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c72000000018' failed - Device or resource busy 2024-02-10T05:53:12.911 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c71600000039' failed - Device or resource busy 2024-02-10T05:53:12.911 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c73a00000019' failed - Device or resource busy 2024-02-10T05:53:12.912 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c71b00000017' failed - Device or resource busy 2024-02-10T05:53:12.943 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c75d0000004c' failed - Device or resource busy 2024-02-10T05:53:12.944 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c74e00000051' failed - Device or resource busy 2024-02-10T05:53:12.954 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c76900000058' failed - Device or resource busy 2024-02-10T05:53:12.954 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c78200000068' failed - Device or resource busy 2024-02-10T05:53:12.955 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c75c00000076' failed - Device or resource busy 2024-02-10T05:53:12.956 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c72d0000004e' failed - Device or resource busy 2024-02-10T05:53:12.956 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c76f00000075' failed - Device or resource busy 2024-02-10T05:53:12.957 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c7520000006a' failed - Device or resource busy 2024-02-10T05:53:12.959 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c73100000074' failed - Device or resource busy 2024-02-10T05:53:12.959 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client3/~dmtmp/PARADOX/.nfs000001000000c76d00000024' failed - Device or resource busy 2024-02-10T05:53:12.959 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c7740000006c' failed - Device or resource busy 2024-02-10T05:53:12.967 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c77d00000066' failed - Device or resource busy 2024-02-10T05:53:12.986 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c7880000006e' failed - Device or resource busy 2024-02-10T05:53:12.986 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c7280000003d' failed - Device or resource busy 2024-02-10T05:53:12.986 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c73b00000064' failed - Device or resource busy 2024-02-10T05:53:12.988 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c74300000054' failed - Device or resource busy 2024-02-10T05:53:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c75700000070' failed - Device or resource busy 2024-02-10T05:53:13.000 INFO:teuthology.orchestra.run.smithi028.stdout:[2046] unlink './clients/client0/~dmtmp/PARADOX/.nfs000001000000c72300000072' failed - Device or resource busy 2024-02-10T05:53:13.030 INFO:teuthology.orchestra.run.smithi028.stdout: 0 cleanup 5 sec 2024-02-10T05:53:13.030 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:53:13.030 INFO:teuthology.orchestra.run.smithi028.stdout: Operation Count AvgLat MaxLat 2024-02-10T05:53:13.031 INFO:teuthology.orchestra.run.smithi028.stdout: ---------------------------------------- 2024-02-10T05:53:13.031 INFO:teuthology.orchestra.run.smithi028.stdout: NTCreateX 1548 4.357 61.389 2024-02-10T05:53:13.031 INFO:teuthology.orchestra.run.smithi028.stdout: Close 1278 5.036 1047.016 2024-02-10T05:53:13.031 INFO:teuthology.orchestra.run.smithi028.stdout: Rename 55 7.447 25.786 2024-02-10T05:53:13.031 INFO:teuthology.orchestra.run.smithi028.stdout: Unlink 126 3.039 15.190 2024-02-10T05:53:13.031 INFO:teuthology.orchestra.run.smithi028.stdout: Qpathinfo 1048 1.408 17.986 2024-02-10T05:53:13.031 INFO:teuthology.orchestra.run.smithi028.stdout: Qfileinfo 502 17.926 1020.968 2024-02-10T05:53:13.031 INFO:teuthology.orchestra.run.smithi028.stdout: Qfsinfo 165 0.579 3.845 2024-02-10T05:53:13.031 INFO:teuthology.orchestra.run.smithi028.stdout: Sfileinfo 170 7.224 55.582 2024-02-10T05:53:13.031 INFO:teuthology.orchestra.run.smithi028.stdout: Find 369 1.033 4.077 2024-02-10T05:53:13.031 INFO:teuthology.orchestra.run.smithi028.stdout: WriteX 2051 0.024 0.063 2024-02-10T05:53:13.031 INFO:teuthology.orchestra.run.smithi028.stdout: ReadX 1523 0.013 0.661 2024-02-10T05:53:13.032 INFO:teuthology.orchestra.run.smithi028.stdout: Flush 144 8.762 20.245 2024-02-10T05:53:13.032 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:53:13.032 INFO:teuthology.orchestra.run.smithi028.stdout:Throughput 22.4682 MB/sec 5 clients 5 procs max_latency=1047.028 ms 2024-02-10T05:53:13.033 DEBUG:teuthology.parallel:result is None 2024-02-10T05:53:13.086 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:53:13.086 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 7 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-10T05:53:13.087 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 11 2024-02-10T05:53:13.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:53:13.088 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:53:13.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:13 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:53:13.065+0000 7f16dacf97c0 -1 Falling back to public interface 2024-02-10T05:53:13.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:13 smithi192 ceph-mon[52881]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:13.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:13 smithi192 ceph-mon[52881]: from='client.44363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:13.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:13 smithi192 ceph-mon[52881]: pgmap v284: 97 pgs: 5 active+undersized, 29 active+undersized+degraded, 8 peering, 55 active+clean; 321 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 512 op/s; 134/1278 objects degraded (10.485%) 2024-02-10T05:53:13.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:13.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:13.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:13.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:13 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/4269772949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:13.430 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:53:13.431 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:53:13.431 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:53:13.431 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:53:13.431 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:53:13.431 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:53:13.431 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:53:13.431 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:53:13.431 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:53:13.431 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "13/23 daemons upgraded", 2024-02-10T05:53:13.431 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading osd daemons", 2024-02-10T05:53:13.431 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:53:13.431 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:53:13.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:13 smithi028 ceph-mon[66336]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:13 smithi028 ceph-mon[66336]: from='client.44363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:13 smithi028 ceph-mon[66336]: pgmap v284: 97 pgs: 5 active+undersized, 29 active+undersized+degraded, 8 peering, 55 active+clean; 321 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 512 op/s; 134/1278 objects degraded (10.485%) 2024-02-10T05:53:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:13 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/4269772949' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Reduced data availability: 1 pg inactive, 1 pg peering; Degraded data redundancy: 134/1278 objects degraded (10.485%), 29 pgs degraded 2024-02-10T05:53:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-10T05:53:13.865 INFO:teuthology.orchestra.run.smithi028.stdout: osd.7 (root=default,host=smithi192) is down 2024-02-10T05:53:13.865 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-02-10T05:53:13.865 INFO:teuthology.orchestra.run.smithi028.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1c is stuck peering for 64s, current state peering, last acting [5,2] 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 134/1278 objects degraded (10.485%), 29 pgs degraded 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.4 is active+undersized+degraded, acting [1,0] 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-02-10T05:53:13.866 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1b is active+undersized+degraded, acting [3,6] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-02-10T05:53:13.867 INFO:teuthology.orchestra.run.smithi028.stdout: pg 4.0 is active+undersized+degraded, acting [6,0] 2024-02-10T05:53:13.868 INFO:teuthology.orchestra.run.smithi028.stdout: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-02-10T05:53:13.868 INFO:teuthology.orchestra.run.smithi028.stdout: pg 4.a is active+undersized+degraded, acting [6,2] 2024-02-10T05:53:13.868 INFO:teuthology.orchestra.run.smithi028.stdout: pg 4.14 is active+undersized+degraded, acting [3,2] 2024-02-10T05:53:14.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:14 smithi028 ceph-mon[66336]: from='client.34438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:14.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:14 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2239676939' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:53:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:14 smithi192 ceph-mon[52881]: from='client.34438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:14 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2239676939' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:53:15.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:15 smithi028 ceph-mon[66336]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:15.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:15 smithi028 ceph-mon[66336]: pgmap v285: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 314 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 14 MiB/s wr, 597 op/s; 165/1293 objects degraded (12.761%) 2024-02-10T05:53:15.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:15 smithi028 ceph-mon[66336]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T05:53:15.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:15.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:53:15.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:15 smithi192 ceph-mon[52881]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:15 smithi192 ceph-mon[52881]: pgmap v285: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 314 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 14 MiB/s wr, 597 op/s; 165/1293 objects degraded (12.761%) 2024-02-10T05:53:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:15 smithi192 ceph-mon[52881]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T05:53:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:53:16.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:16 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 188/1374 objects degraded (13.683%), 35 pgs degraded (PG_DEGRADED) 2024-02-10T05:53:16.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:16 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 188/1374 objects degraded (13.683%), 35 pgs degraded (PG_DEGRADED) 2024-02-10T05:53:17.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:17 smithi028 ceph-mon[66336]: pgmap v286: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 304 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s wr, 452 op/s; 188/1374 objects degraded (13.683%) 2024-02-10T05:53:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:17 smithi192 ceph-mon[52881]: pgmap v286: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 304 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 14 MiB/s wr, 452 op/s; 188/1374 objects degraded (13.683%) 2024-02-10T05:53:19.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:18 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:19.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:18 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:19.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:18 smithi028 ceph-mon[66336]: pgmap v287: 97 pgs: 8 active+undersized, 34 active+undersized+degraded, 55 active+clean; 282 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 12 MiB/s wr, 444 op/s; 170/1086 objects degraded (15.654%) 2024-02-10T05:53:19.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:18 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:19.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:18 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:19.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:18 smithi192 ceph-mon[52881]: pgmap v287: 97 pgs: 8 active+undersized, 34 active+undersized+degraded, 55 active+clean; 282 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 12 MiB/s wr, 444 op/s; 170/1086 objects degraded (15.654%) 2024-02-10T05:53:20.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:19 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:20.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:19 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:20.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:19 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:20.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:19 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:20.779 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:20 smithi192 ceph-mon[52881]: pgmap v288: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 249 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 10 MiB/s wr, 384 op/s; 107/705 objects degraded (15.177%) 2024-02-10T05:53:21.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:20 smithi028 ceph-mon[66336]: pgmap v288: 97 pgs: 9 active+undersized, 33 active+undersized+degraded, 55 active+clean; 249 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 10 MiB/s wr, 384 op/s; 107/705 objects degraded (15.177%) 2024-02-10T05:53:21.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:20 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:53:20.986+0000 7f16dacf97c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-02-10T05:53:22.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:22 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 72/420 objects degraded (17.143%), 31 pgs degraded (PG_DEGRADED) 2024-02-10T05:53:22.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:22 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 72/420 objects degraded (17.143%), 31 pgs degraded (PG_DEGRADED) 2024-02-10T05:53:22.822 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:22 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:53:22.616+0000 7f16dacf97c0 -1 osd.7 152 log_to_monitors true 2024-02-10T05:53:23.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: pgmap v289: 97 pgs: 11 active+undersized, 31 active+undersized+degraded, 55 active+clean; 217 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 8.8 MiB/s wr, 366 op/s; 72/420 objects degraded (17.143%) 2024-02-10T05:53:23.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:23.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:23.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:23.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:23.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:23.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:23.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:23.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:23.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:23.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:23.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:23.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:23.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T05:53:23.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-10T05:53:23.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T05:53:23.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-10T05:53:23.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T05:53:23.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-10T05:53:23.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T05:53:23.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-10T05:53:23.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T05:53:23.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-10T05:53:23.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T05:53:23.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-10T05:53:23.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T05:53:23.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-10T05:53:23.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T05:53:23.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-10T05:53:23.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T05:53:23.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='osd.7 [v2:172.21.15.192:6824/1170198165,v1:172.21.15.192:6825/1170198165]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:53:23.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:23 smithi028 ceph-mon[66336]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:53:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: pgmap v289: 97 pgs: 11 active+undersized, 31 active+undersized+degraded, 55 active+clean; 217 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 8.8 MiB/s wr, 366 op/s; 72/420 objects degraded (17.143%) 2024-02-10T05:53:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-10T05:53:23.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-10T05:53:23.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-10T05:53:23.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-10T05:53:23.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-10T05:53:23.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-10T05:53:23.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-10T05:53:23.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-10T05:53:23.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-10T05:53:23.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-10T05:53:23.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-10T05:53:23.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='osd.7 [v2:172.21.15.192:6824/1170198165,v1:172.21.15.192:6825/1170198165]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:53:23.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-mon[52881]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:53:24.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:53:23 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:53:23.627+0000 7f16cc151700 -1 osd.7 152 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:53:24.475 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:24 smithi028 ceph-mon[66336]: Upgrade: Setting container_image for all osd 2024-02-10T05:53:24.475 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:24 smithi028 ceph-mon[66336]: Upgrade: Setting require_osd_release to 19 squid 2024-02-10T05:53:24.475 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:24 smithi028 ceph-mon[66336]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-10T05:53:24.475 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:24 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-10T05:53:24.475 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:24 smithi028 ceph-mon[66336]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:53:24.475 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:24 smithi028 ceph-mon[66336]: osdmap e155: 8 total, 7 up, 8 in 2024-02-10T05:53:24.475 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:24 smithi028 ceph-mon[66336]: from='osd.7 [v2:172.21.15.192:6824/1170198165,v1:172.21.15.192:6825/1170198165]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:53:24.475 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:24 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi028.xhtowe"]}]: dispatch 2024-02-10T05:53:24.475 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:24 smithi028 ceph-mon[66336]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:53:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:24 smithi192 ceph-mon[52881]: Upgrade: Setting container_image for all osd 2024-02-10T05:53:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:24 smithi192 ceph-mon[52881]: Upgrade: Setting require_osd_release to 19 squid 2024-02-10T05:53:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:24 smithi192 ceph-mon[52881]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-10T05:53:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:24 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-10T05:53:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:24 smithi192 ceph-mon[52881]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:53:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:24 smithi192 ceph-mon[52881]: osdmap e155: 8 total, 7 up, 8 in 2024-02-10T05:53:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:24 smithi192 ceph-mon[52881]: from='osd.7 [v2:172.21.15.192:6824/1170198165,v1:172.21.15.192:6825/1170198165]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:53:24.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:24 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi028.xhtowe"]}]: dispatch 2024-02-10T05:53:24.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:24 smithi192 ceph-mon[52881]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-02-10T05:53:25.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:25 smithi192 ceph-mon[52881]: Upgrade: It appears safe to stop mds.foofs.smithi028.xhtowe 2024-02-10T05:53:25.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:25 smithi192 ceph-mon[52881]: pgmap v291: 97 pgs: 11 active+undersized, 31 active+undersized+degraded, 55 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.3 MiB/s wr, 131 op/s; 72/420 objects degraded (17.143%) 2024-02-10T05:53:25.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:25 smithi192 ceph-mon[52881]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:53:25.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:25 smithi192 ceph-mon[52881]: osd.7 [v2:172.21.15.192:6824/1170198165,v1:172.21.15.192:6825/1170198165] boot 2024-02-10T05:53:25.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:25 smithi192 ceph-mon[52881]: osdmap e156: 8 total, 8 up, 8 in 2024-02-10T05:53:25.474 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:53:25.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:25 smithi028 ceph-mon[66336]: Upgrade: It appears safe to stop mds.foofs.smithi028.xhtowe 2024-02-10T05:53:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:25 smithi028 ceph-mon[66336]: pgmap v291: 97 pgs: 11 active+undersized, 31 active+undersized+degraded, 55 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.3 MiB/s wr, 131 op/s; 72/420 objects degraded (17.143%) 2024-02-10T05:53:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:25 smithi028 ceph-mon[66336]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T05:53:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:25 smithi028 ceph-mon[66336]: osd.7 [v2:172.21.15.192:6824/1170198165,v1:172.21.15.192:6825/1170198165] boot 2024-02-10T05:53:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:25 smithi028 ceph-mon[66336]: osdmap e156: 8 total, 8 up, 8 in 2024-02-10T05:53:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:53:26.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:26 smithi028 ceph-mon[66336]: Upgrade: Updating mds.foofs.smithi028.xhtowe 2024-02-10T05:53:26.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:26 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:26.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:26 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi028.xhtowe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T05:53:26.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:26 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:26.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:26 smithi028 ceph-mon[66336]: Deploying daemon mds.foofs.smithi028.xhtowe on smithi028 2024-02-10T05:53:26.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:26 smithi028 ceph-mon[66336]: osdmap e157: 8 total, 8 up, 8 in 2024-02-10T05:53:26.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:26 smithi028 ceph-mon[66336]: pgmap v294: 97 pgs: 13 peering, 9 active+undersized, 20 active+undersized+degraded, 55 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 43 KiB/s wr, 16 op/s; 56/420 objects degraded (13.333%) 2024-02-10T05:53:26.737 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:26 smithi192 ceph-mon[52881]: Upgrade: Updating mds.foofs.smithi028.xhtowe 2024-02-10T05:53:26.737 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:26 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:26.737 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:26 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi028.xhtowe", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T05:53:26.737 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:26 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:26.737 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:26 smithi192 ceph-mon[52881]: Deploying daemon mds.foofs.smithi028.xhtowe on smithi028 2024-02-10T05:53:26.737 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:26 smithi192 ceph-mon[52881]: osdmap e157: 8 total, 8 up, 8 in 2024-02-10T05:53:26.737 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:26 smithi192 ceph-mon[52881]: pgmap v294: 97 pgs: 13 peering, 9 active+undersized, 20 active+undersized+degraded, 55 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 43 KiB/s wr, 16 op/s; 56/420 objects degraded (13.333%) 2024-02-10T05:53:28.043 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:27 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028[66332]: 2024-02-10T05:53:27.609+0000 7f398c7c0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T05:53:28.043 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:27 smithi028 ceph-mon[66336]: osdmap e158: 8 total, 8 up, 8 in 2024-02-10T05:53:28.043 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:27 smithi028 ceph-mon[66336]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-10T05:53:28.044 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:27 smithi028 ceph-mon[66336]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T05:53:28.044 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:27 smithi028 ceph-mon[66336]: osdmap e159: 8 total, 8 up, 8 in 2024-02-10T05:53:28.044 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:27 smithi028 ceph-mon[66336]: Standby daemon mds.foofs.smithi192.hdcbzw assigned to filesystem foofs as rank 0 2024-02-10T05:53:28.044 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:27 smithi028 ceph-mon[66336]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-10T05:53:28.044 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:27 smithi028 ceph-mon[66336]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T05:53:28.044 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:27 smithi028 ceph-mon[66336]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-02-10T05:53:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:27 smithi192 ceph-mon[52881]: osdmap e158: 8 total, 8 up, 8 in 2024-02-10T05:53:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:27 smithi192 ceph-mon[52881]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-10T05:53:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:27 smithi192 ceph-mon[52881]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T05:53:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:27 smithi192 ceph-mon[52881]: osdmap e159: 8 total, 8 up, 8 in 2024-02-10T05:53:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:27 smithi192 ceph-mon[52881]: Standby daemon mds.foofs.smithi192.hdcbzw assigned to filesystem foofs as rank 0 2024-02-10T05:53:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:27 smithi192 ceph-mon[52881]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-10T05:53:28.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:27 smithi192 ceph-mon[52881]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T05:53:28.073 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:27 smithi192 ceph-mon[52881]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-02-10T05:53:29.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:28 smithi028 ceph-mon[66336]: fsmap foofs:1/1 {0=foofs.smithi192.hdcbzw=up:replay} 2024-02-10T05:53:29.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:28 smithi028 ceph-mon[66336]: Health check update: Degraded data redundancy: 56/420 objects degraded (13.333%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T05:53:29.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:28 smithi028 ceph-mon[66336]: pgmap v297: 97 pgs: 1 remapped+peering, 14 peering, 6 active+undersized, 12 active+undersized+degraded, 64 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 767 B/s wr, 0 op/s; 34/420 objects degraded (8.095%); 1024 KiB/s, 5 objects/s recovering 2024-02-10T05:53:29.059 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:53:28 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:53:28.647+0000 7f47a7a08700 -1 bdev(0x55b1b22bc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 17 2024-02-10T05:53:29.059 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:53:28 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:53:28.647+0000 7f47a8209700 -1 bdev(0x55b1b22bc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 19 2024-02-10T05:53:29.059 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:53:28 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:53:28.647+0000 7f47a920b700 -1 bdev(0x55b1b22bc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 19 2024-02-10T05:53:29.059 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:53:28 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:53:28.647+0000 7f47ab20f700 -1 bdev(0x55b1b22bc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 18 2024-02-10T05:53:29.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:28 smithi192 ceph-mon[52881]: fsmap foofs:1/1 {0=foofs.smithi192.hdcbzw=up:replay} 2024-02-10T05:53:29.071 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:28 smithi192 ceph-mon[52881]: Health check update: Degraded data redundancy: 56/420 objects degraded (13.333%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T05:53:29.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:28 smithi192 ceph-mon[52881]: pgmap v297: 97 pgs: 1 remapped+peering, 14 peering, 6 active+undersized, 12 active+undersized+degraded, 64 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 767 B/s wr, 0 op/s; 34/420 objects degraded (8.095%); 1024 KiB/s, 5 objects/s recovering 2024-02-10T05:53:30.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:29 smithi028 ceph-mon[66336]: osdmap e160: 8 total, 8 up, 8 in 2024-02-10T05:53:30.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:29 smithi192 ceph-mon[52881]: osdmap e160: 8 total, 8 up, 8 in 2024-02-10T05:53:31.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:31 smithi028 ceph-mon[66336]: pgmap v299: 97 pgs: 1 remapped+peering, 14 peering, 82 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 693 B/s wr, 1 op/s; 925 KiB/s, 51 objects/s recovering 2024-02-10T05:53:31.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:31.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:53:31.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:31 smithi028 ceph-mon[66336]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/420 objects degraded (8.095%), 12 pgs degraded) 2024-02-10T05:53:31.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:31 smithi192 ceph-mon[52881]: pgmap v299: 97 pgs: 1 remapped+peering, 14 peering, 82 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 693 B/s wr, 1 op/s; 925 KiB/s, 51 objects/s recovering 2024-02-10T05:53:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:53:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:31 smithi192 ceph-mon[52881]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/420 objects degraded (8.095%), 12 pgs degraded) 2024-02-10T05:53:33.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:32 smithi028 ceph-mon[66336]: reconnect by client.24339 172.21.15.192:0/2291645479 after 0 2024-02-10T05:53:33.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:32 smithi028 ceph-mon[66336]: reconnect by client.14542 172.21.15.28:0/1798128558 after 0 2024-02-10T05:53:33.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:32 smithi028 ceph-mon[66336]: mds.? [v2:172.21.15.192:6832/853070905,v1:172.21.15.192:6833/853070905] up:reconnect 2024-02-10T05:53:33.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:32 smithi028 ceph-mon[66336]: fsmap foofs:1/1 {0=foofs.smithi192.hdcbzw=up:reconnect} 2024-02-10T05:53:33.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:32 smithi028 ceph-mon[66336]: pgmap v300: 97 pgs: 1 peering, 96 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 511 B/s wr, 5 op/s; 1.2 MiB/s, 56 objects/s recovering 2024-02-10T05:53:33.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:32 smithi192 ceph-mon[52881]: reconnect by client.24339 172.21.15.192:0/2291645479 after 0 2024-02-10T05:53:33.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:32 smithi192 ceph-mon[52881]: reconnect by client.14542 172.21.15.28:0/1798128558 after 0 2024-02-10T05:53:33.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:32 smithi192 ceph-mon[52881]: mds.? [v2:172.21.15.192:6832/853070905,v1:172.21.15.192:6833/853070905] up:reconnect 2024-02-10T05:53:33.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:32 smithi192 ceph-mon[52881]: fsmap foofs:1/1 {0=foofs.smithi192.hdcbzw=up:reconnect} 2024-02-10T05:53:33.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:32 smithi192 ceph-mon[52881]: pgmap v300: 97 pgs: 1 peering, 96 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 511 B/s wr, 5 op/s; 1.2 MiB/s, 56 objects/s recovering 2024-02-10T05:53:34.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:33 smithi028 ceph-mon[66336]: mds.? [v2:172.21.15.192:6832/853070905,v1:172.21.15.192:6833/853070905] up:rejoin 2024-02-10T05:53:34.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:33 smithi028 ceph-mon[66336]: fsmap foofs:1/1 {0=foofs.smithi192.hdcbzw=up:rejoin} 2024-02-10T05:53:34.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:33 smithi028 ceph-mon[66336]: daemon mds.foofs.smithi192.hdcbzw is now active in filesystem foofs as rank 0 2024-02-10T05:53:34.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:33 smithi192 ceph-mon[52881]: mds.? [v2:172.21.15.192:6832/853070905,v1:172.21.15.192:6833/853070905] up:rejoin 2024-02-10T05:53:34.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:33 smithi192 ceph-mon[52881]: fsmap foofs:1/1 {0=foofs.smithi192.hdcbzw=up:rejoin} 2024-02-10T05:53:34.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:33 smithi192 ceph-mon[52881]: daemon mds.foofs.smithi192.hdcbzw is now active in filesystem foofs as rank 0 2024-02-10T05:53:35.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:34 smithi028 ceph-mon[66336]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-10T05:53:35.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:34 smithi028 ceph-mon[66336]: mds.? [v2:172.21.15.192:6832/853070905,v1:172.21.15.192:6833/853070905] up:active 2024-02-10T05:53:35.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:34 smithi028 ceph-mon[66336]: fsmap foofs:1 {0=foofs.smithi192.hdcbzw=up:active} 2024-02-10T05:53:35.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:34 smithi028 ceph-mon[66336]: pgmap v301: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 7 op/s; 1005 KiB/s, 45 objects/s recovering 2024-02-10T05:53:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:34 smithi192 ceph-mon[52881]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-10T05:53:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:34 smithi192 ceph-mon[52881]: mds.? [v2:172.21.15.192:6832/853070905,v1:172.21.15.192:6833/853070905] up:active 2024-02-10T05:53:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:34 smithi192 ceph-mon[52881]: fsmap foofs:1 {0=foofs.smithi192.hdcbzw=up:active} 2024-02-10T05:53:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:34 smithi192 ceph-mon[52881]: pgmap v301: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 7 op/s; 1005 KiB/s, 45 objects/s recovering 2024-02-10T05:53:37.267 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:37 smithi028 ceph-mon[66336]: pgmap v302: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 10 op/s; 885 KiB/s, 40 objects/s recovering 2024-02-10T05:53:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:37 smithi192 ceph-mon[52881]: pgmap v302: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 10 op/s; 885 KiB/s, 40 objects/s recovering 2024-02-10T05:53:38.821 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:38 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:38.821 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:38 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:38.821 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:38 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:38.821 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:38 smithi028 ceph-mon[66336]: pgmap v303: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 8.6 KiB/s wr, 13 op/s; 336 KiB/s, 31 objects/s recovering 2024-02-10T05:53:38.821 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:38 smithi028 ceph-mon[66336]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-10T05:53:38.821 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:38 smithi028 ceph-mon[66336]: Cluster is now healthy 2024-02-10T05:53:38.821 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:38 smithi028 ceph-mon[66336]: mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472] up:boot 2024-02-10T05:53:38.821 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:38 smithi028 ceph-mon[66336]: fsmap foofs:1 {0=foofs.smithi192.hdcbzw=up:active} 1 up:standby 2024-02-10T05:53:38.821 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:38 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi028.xhtowe"}]: dispatch 2024-02-10T05:53:38.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:38 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:38.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:38 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:38.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:38 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:38.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:38 smithi192 ceph-mon[52881]: pgmap v303: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 8.6 KiB/s wr, 13 op/s; 336 KiB/s, 31 objects/s recovering 2024-02-10T05:53:38.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:38 smithi192 ceph-mon[52881]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-10T05:53:38.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:38 smithi192 ceph-mon[52881]: Cluster is now healthy 2024-02-10T05:53:38.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:38 smithi192 ceph-mon[52881]: mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472] up:boot 2024-02-10T05:53:38.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:38 smithi192 ceph-mon[52881]: fsmap foofs:1 {0=foofs.smithi192.hdcbzw=up:active} 1 up:standby 2024-02-10T05:53:38.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:38 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi028.xhtowe"}]: dispatch 2024-02-10T05:53:41.448 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:41 smithi028 ceph-mon[66336]: pgmap v304: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 7.5 KiB/s wr, 12 op/s; 295 KiB/s, 14 objects/s recovering 2024-02-10T05:53:41.449 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:41 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:41 smithi192 ceph-mon[52881]: pgmap v304: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 7.5 KiB/s wr, 12 op/s; 295 KiB/s, 14 objects/s recovering 2024-02-10T05:53:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:41 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:43.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:43 smithi028 ceph-mon[66336]: pgmap v305: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 7.2 KiB/s wr, 11 op/s; 280 KiB/s, 9 objects/s recovering 2024-02-10T05:53:43.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:43.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:43 smithi192 ceph-mon[52881]: pgmap v305: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 7.2 KiB/s wr, 11 op/s; 280 KiB/s, 9 objects/s recovering 2024-02-10T05:53:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:44.233 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:53:44.947 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:53:44.947 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (9m) 2s ago 19m 23.8M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:53:44.947 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (5m) 2s ago 19m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:53:44.947 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (5m) 27s ago 18m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:53:44.947 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (9m) 2s ago 18m 45.9M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:53:44.947 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (8s) 2s ago 13m 11.1M - 19.0.0-1361-g290212a1 c328ab9dcfc7 91720c6bc17a 2024-02-10T05:53:44.947 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (13m) 27s ago 13m 15.4M - 17.2.0 e1d6a67b021e f34470633a52 2024-02-10T05:53:44.947 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (8m) 2s ago 20m 510M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:53:44.947 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (7m) 27s ago 18m 454M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:53:44.947 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (7m) 2s ago 20m 59.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (6m) 27s ago 18m 45.0M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (12m) 2s ago 12m 277M - 4.0 e1d6a67b021e 5492a70ebf64 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (12m) 27s ago 12m 86.3M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (9m) 2s ago 18m 9801k - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (8m) 27s ago 18m 18.0M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (5m) 2s ago 17m 406M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (4m) 2s ago 17m 369M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (3m) 2s ago 16m 353M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 a45f0e24ae44 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (2m) 2s ago 16m 393M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 25f4cf94969e 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (2m) 27s ago 15m 340M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 b15b2733ec77 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (108s) 27s ago 15m 308M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 000913de61d6 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (69s) 27s ago 15m 280M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 c9776c8b4f21 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (32s) 27s ago 14m 14.7M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 86dbef76e22d 2024-02-10T05:53:44.948 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (8m) 2s ago 17m 65.6M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:53:45.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:44 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:45.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:44 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:45.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:44 smithi028 ceph-mon[66336]: pgmap v306: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-02-10T05:53:45.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:44 smithi028 ceph-mon[66336]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:45.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:44 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:45.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:44 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:45.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:44 smithi192 ceph-mon[52881]: pgmap v306: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-02-10T05:53:45.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:44 smithi192 ceph-mon[52881]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:53:45.363 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:53:45.363 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:53:45.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:53:45.363 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:53:45.363 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 1 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 13 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:53:45.364 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:53:45.714 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:53:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:53:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:53:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:53:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:53:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:53:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "osd", 2024-02-10T05:53:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:53:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:53:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:53:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "15/23 daemons upgraded", 2024-02-10T05:53:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading mds daemons", 2024-02-10T05:53:45.715 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:53:45.715 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:53:46.137 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:53:46.240 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:46 smithi028 ceph-mon[66336]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:46.240 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:46 smithi028 ceph-mon[66336]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:46.240 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:46.240 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:53:46.241 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:46 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2237003816' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:46 smithi192 ceph-mon[52881]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:46 smithi192 ceph-mon[52881]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:53:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:46 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2237003816' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:47.140 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:47 smithi028 ceph-mon[66336]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:47.140 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:47 smithi028 ceph-mon[66336]: pgmap v307: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 7.2 KiB/s wr, 7 op/s 2024-02-10T05:53:47.140 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:47 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/856541560' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:53:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:47 smithi192 ceph-mon[52881]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:47 smithi192 ceph-mon[52881]: pgmap v307: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s rd, 7.2 KiB/s wr, 7 op/s 2024-02-10T05:53:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:47 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/856541560' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:53:48.983 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: Detected new or changed devices on smithi028 2024-02-10T05:53:48.983 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi192.hdcbzw"]}]: dispatch 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: Upgrade: It appears safe to stop mds.foofs.smithi192.hdcbzw 2024-02-10T05:53:48.984 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:48 smithi192 ceph-mon[52881]: pgmap v308: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 7.3 KiB/s wr, 4 op/s 2024-02-10T05:53:49.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: Detected new or changed devices on smithi028 2024-02-10T05:53:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:53:49.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi192.hdcbzw"]}]: dispatch 2024-02-10T05:53:49.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: Upgrade: It appears safe to stop mds.foofs.smithi192.hdcbzw 2024-02-10T05:53:49.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:48 smithi028 ceph-mon[66336]: pgmap v308: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 7.3 KiB/s wr, 4 op/s 2024-02-10T05:53:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:50 smithi192 ceph-mon[52881]: Upgrade: Updating mds.foofs.smithi192.hdcbzw 2024-02-10T05:53:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:50 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:50 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi192.hdcbzw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T05:53:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:50 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:50 smithi192 ceph-mon[52881]: Deploying daemon mds.foofs.smithi192.hdcbzw on smithi192 2024-02-10T05:53:50.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:50 smithi192 ceph-mon[52881]: pgmap v309: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 937 B/s rd, 170 B/s wr, 1 op/s 2024-02-10T05:53:51.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:50 smithi028 ceph-mon[66336]: Upgrade: Updating mds.foofs.smithi192.hdcbzw 2024-02-10T05:53:51.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:50 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:53:51.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:50 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi192.hdcbzw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-10T05:53:51.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:50 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:51.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:50 smithi028 ceph-mon[66336]: Deploying daemon mds.foofs.smithi192.hdcbzw on smithi192 2024-02-10T05:53:51.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:50 smithi028 ceph-mon[66336]: pgmap v309: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 937 B/s rd, 170 B/s wr, 1 op/s 2024-02-10T05:53:52.308 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:53:51 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:53:51.912+0000 7f47ad213700 -1 bdev(0x55b1b22bc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 19 2024-02-10T05:53:52.309 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:53:51 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:53:51.912+0000 7f47aba10700 -1 bdev(0x55b1b22bc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 24 2024-02-10T05:53:52.309 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:53:51 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:53:51.912+0000 7f47a7a08700 -1 bdev(0x55b1b22bc000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 14 2024-02-10T05:53:52.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:51 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028[66332]: 2024-02-10T05:53:51.888+0000 7f398c7c0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T05:53:52.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:51 smithi028 ceph-mon[66336]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-10T05:53:52.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:51 smithi028 ceph-mon[66336]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T05:53:52.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:51 smithi028 ceph-mon[66336]: osdmap e161: 8 total, 8 up, 8 in 2024-02-10T05:53:52.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:51 smithi028 ceph-mon[66336]: Standby daemon mds.foofs.smithi028.xhtowe assigned to filesystem foofs as rank 0 2024-02-10T05:53:52.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:51 smithi028 ceph-mon[66336]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-10T05:53:52.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:51 smithi028 ceph-mon[66336]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T05:53:52.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:51 smithi028 ceph-mon[66336]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-02-10T05:53:52.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:51 smithi028 ceph-mon[66336]: fsmap foofs:1/1 {0=foofs.smithi028.xhtowe=up:replay} 2024-02-10T05:53:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:51 smithi192 ceph-mon[52881]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-10T05:53:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:51 smithi192 ceph-mon[52881]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-10T05:53:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:51 smithi192 ceph-mon[52881]: osdmap e161: 8 total, 8 up, 8 in 2024-02-10T05:53:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:51 smithi192 ceph-mon[52881]: Standby daemon mds.foofs.smithi028.xhtowe assigned to filesystem foofs as rank 0 2024-02-10T05:53:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:51 smithi192 ceph-mon[52881]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-10T05:53:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:51 smithi192 ceph-mon[52881]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-10T05:53:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:51 smithi192 ceph-mon[52881]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-02-10T05:53:52.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:51 smithi192 ceph-mon[52881]: fsmap foofs:1/1 {0=foofs.smithi028.xhtowe=up:replay} 2024-02-10T05:53:53.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:52 smithi028 ceph-mon[66336]: pgmap v311: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 204 B/s wr, 0 op/s 2024-02-10T05:53:53.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:52 smithi192 ceph-mon[52881]: pgmap v311: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 204 B/s wr, 0 op/s 2024-02-10T05:53:54.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:53 smithi028 ceph-mon[66336]: reconnect by client.14542 172.21.15.28:0/1798128558 after 0 2024-02-10T05:53:54.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:53 smithi028 ceph-mon[66336]: mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472] up:reconnect 2024-02-10T05:53:54.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:53 smithi028 ceph-mon[66336]: fsmap foofs:1/1 {0=foofs.smithi028.xhtowe=up:reconnect} 2024-02-10T05:53:54.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:53 smithi028 ceph-mon[66336]: reconnect by client.24339 172.21.15.192:0/2291645479 after 0.0229994 2024-02-10T05:53:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:53 smithi192 ceph-mon[52881]: reconnect by client.14542 172.21.15.28:0/1798128558 after 0 2024-02-10T05:53:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:53 smithi192 ceph-mon[52881]: mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472] up:reconnect 2024-02-10T05:53:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:53 smithi192 ceph-mon[52881]: fsmap foofs:1/1 {0=foofs.smithi028.xhtowe=up:reconnect} 2024-02-10T05:53:54.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:53 smithi192 ceph-mon[52881]: reconnect by client.24339 172.21.15.192:0/2291645479 after 0.0229994 2024-02-10T05:53:55.243 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:54 smithi192 ceph-mon[52881]: mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472] up:rejoin 2024-02-10T05:53:55.243 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:54 smithi192 ceph-mon[52881]: fsmap foofs:1/1 {0=foofs.smithi028.xhtowe=up:rejoin} 2024-02-10T05:53:55.243 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:54 smithi192 ceph-mon[52881]: daemon mds.foofs.smithi028.xhtowe is now active in filesystem foofs as rank 0 2024-02-10T05:53:55.243 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:54 smithi192 ceph-mon[52881]: pgmap v312: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.8 MiB/s rd, 204 B/s wr, 3 op/s 2024-02-10T05:53:55.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:54 smithi028 ceph-mon[66336]: mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472] up:rejoin 2024-02-10T05:53:55.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:54 smithi028 ceph-mon[66336]: fsmap foofs:1/1 {0=foofs.smithi028.xhtowe=up:rejoin} 2024-02-10T05:53:55.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:54 smithi028 ceph-mon[66336]: daemon mds.foofs.smithi028.xhtowe is now active in filesystem foofs as rank 0 2024-02-10T05:53:55.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:54 smithi028 ceph-mon[66336]: pgmap v312: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.8 MiB/s rd, 204 B/s wr, 3 op/s 2024-02-10T05:53:56.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:55 smithi028 ceph-mon[66336]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-10T05:53:56.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:55 smithi028 ceph-mon[66336]: mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472] up:active 2024-02-10T05:53:56.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:55 smithi028 ceph-mon[66336]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 2024-02-10T05:53:56.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:55 smithi192 ceph-mon[52881]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-10T05:53:56.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:55 smithi192 ceph-mon[52881]: mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472] up:active 2024-02-10T05:53:56.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:55 smithi192 ceph-mon[52881]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 2024-02-10T05:53:57.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:56 smithi028 ceph-mon[66336]: pgmap v313: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 204 B/s wr, 7 op/s 2024-02-10T05:53:57.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:56 smithi192 ceph-mon[52881]: pgmap v313: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 204 B/s wr, 7 op/s 2024-02-10T05:53:58.980 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:53:58 smithi192 ceph-mon[52881]: pgmap v314: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 9 op/s 2024-02-10T05:53:59.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:53:58 smithi028 ceph-mon[66336]: pgmap v314: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 9 op/s 2024-02-10T05:54:01.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:01 smithi028 ceph-mon[66336]: pgmap v315: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 10 op/s 2024-02-10T05:54:01.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:01.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:54:01.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:01.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:01.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:01 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:01 smithi192 ceph-mon[52881]: pgmap v315: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 204 B/s wr, 10 op/s 2024-02-10T05:54:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:54:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:01.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:01 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:02.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:02 smithi028 ceph-mon[66336]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-10T05:54:02.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:02 smithi028 ceph-mon[66336]: Cluster is now healthy 2024-02-10T05:54:02.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:02 smithi028 ceph-mon[66336]: mds.? [v2:172.21.15.192:6832/4235903261,v1:172.21.15.192:6833/4235903261] up:boot 2024-02-10T05:54:02.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:02 smithi028 ceph-mon[66336]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 1 up:standby 2024-02-10T05:54:02.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:02 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi192.hdcbzw"}]: dispatch 2024-02-10T05:54:02.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:02 smithi192 ceph-mon[52881]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-10T05:54:02.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:02 smithi192 ceph-mon[52881]: Cluster is now healthy 2024-02-10T05:54:02.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:02 smithi192 ceph-mon[52881]: mds.? [v2:172.21.15.192:6832/4235903261,v1:172.21.15.192:6833/4235903261] up:boot 2024-02-10T05:54:02.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:02 smithi192 ceph-mon[52881]: fsmap foofs:1 {0=foofs.smithi028.xhtowe=up:active} 1 up:standby 2024-02-10T05:54:02.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:02 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi192.hdcbzw"}]: dispatch 2024-02-10T05:54:03.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:03 smithi028 ceph-mon[66336]: pgmap v316: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 7.2 KiB/s wr, 13 op/s 2024-02-10T05:54:03.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:03 smithi192 ceph-mon[52881]: pgmap v316: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 7.2 KiB/s wr, 13 op/s 2024-02-10T05:54:04.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:04.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:04 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:04.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:04 smithi028 ceph-mon[66336]: pgmap v317: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 6.1 KiB/s wr, 11 op/s 2024-02-10T05:54:04.821 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:04.821 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:04 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:04.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:04 smithi192 ceph-mon[52881]: pgmap v317: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 6.1 KiB/s wr, 11 op/s 2024-02-10T05:54:05.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:05 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:05.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:05 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:05.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:05 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:05.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:05 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:06.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:06 smithi028 ceph-mon[66336]: pgmap v318: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 6.1 KiB/s wr, 9 op/s 2024-02-10T05:54:06.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:06 smithi192 ceph-mon[52881]: pgmap v318: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 6.1 KiB/s wr, 9 op/s 2024-02-10T05:54:09.029 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: Detected new or changed devices on smithi192 2024-02-10T05:54:09.029 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: pgmap v319: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 7.3 KiB/s wr, 5 op/s 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: Upgrade: Setting container_image for all mds 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi028.xhtowe"}]: dispatch 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi028.xhtowe"}]': finished 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi192.hdcbzw"}]: dispatch 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi192.hdcbzw"}]': finished 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: Upgrade: Setting container_image for all rgw 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.031 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T05:54:09.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: Upgrade: Setting container_image for all iscsi 2024-02-10T05:54:09.032 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: Detected new or changed devices on smithi192 2024-02-10T05:54:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: pgmap v319: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 7.3 KiB/s wr, 5 op/s 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: Upgrade: Setting container_image for all mds 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi028.xhtowe"}]: dispatch 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi028.xhtowe"}]': finished 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi192.hdcbzw"}]: dispatch 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi192.hdcbzw"}]': finished 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: Upgrade: Setting container_image for all rgw 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: Upgrade: Setting container_image for all rbd-mirror 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: Upgrade: Setting container_image for all ceph-exporter 2024-02-10T05:54:09.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:09.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:09.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: Upgrade: Setting container_image for all iscsi 2024-02-10T05:54:09.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:11.274 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: Upgrade: Updating nfs.foo.0.0.smithi028.tzitkz (1/2) 2024-02-10T05:54:11.274 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:11.274 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: Creating key for client.nfs.foo.0.0.smithi028.tzitkz 2024-02-10T05:54:11.274 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi028.tzitkz", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T05:54:11.274 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-02-10T05:54:11.274 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T05:54:11.274 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T05:54:11.274 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:11.274 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T05:54:11.274 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T05:54:11.275 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: Rados config object exists: conf-nfs.foo 2024-02-10T05:54:11.275 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: Creating key for client.nfs.foo.0.0.smithi028.tzitkz-rgw 2024-02-10T05:54:11.275 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi028.tzitkz-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:54:11.275 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: Bind address in nfs.foo.0.0.smithi028.tzitkz's ganesha conf is defaulting to empty 2024-02-10T05:54:11.275 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:11.275 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: Deploying daemon nfs.foo.0.0.smithi028.tzitkz on smithi028 2024-02-10T05:54:11.275 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:10 smithi028 ceph-mon[66336]: pgmap v320: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 7.2 KiB/s wr, 3 op/s 2024-02-10T05:54:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: Upgrade: Updating nfs.foo.0.0.smithi028.tzitkz (1/2) 2024-02-10T05:54:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: Creating key for client.nfs.foo.0.0.smithi028.tzitkz 2024-02-10T05:54:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi028.tzitkz", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T05:54:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-02-10T05:54:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T05:54:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T05:54:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T05:54:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T05:54:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: Rados config object exists: conf-nfs.foo 2024-02-10T05:54:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: Creating key for client.nfs.foo.0.0.smithi028.tzitkz-rgw 2024-02-10T05:54:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi028.tzitkz-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:54:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: Bind address in nfs.foo.0.0.smithi028.tzitkz's ganesha conf is defaulting to empty 2024-02-10T05:54:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: Deploying daemon nfs.foo.0.0.smithi028.tzitkz on smithi028 2024-02-10T05:54:11.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:10 smithi192 ceph-mon[52881]: pgmap v320: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 7.2 KiB/s wr, 3 op/s 2024-02-10T05:54:13.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:13 smithi028 ceph-mon[66336]: pgmap v321: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 7.3 KiB/s wr, 3 op/s 2024-02-10T05:54:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:13 smithi192 ceph-mon[52881]: pgmap v321: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 7.3 KiB/s wr, 3 op/s 2024-02-10T05:54:14.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:14 smithi028 ceph-mon[66336]: from='mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472]' entity='mds.foofs.smithi028.xhtowe' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.28:0/1798128558"}]: dispatch 2024-02-10T05:54:14.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:14 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:14.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:14 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:14.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:14 smithi192 ceph-mon[52881]: from='mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472]' entity='mds.foofs.smithi028.xhtowe' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.28:0/1798128558"}]: dispatch 2024-02-10T05:54:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:14 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:14 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:15 smithi192 ceph-mon[52881]: Evicting (and blocklisting) client session 14542 (172.21.15.28:0/1798128558) 2024-02-10T05:54:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:15 smithi192 ceph-mon[52881]: pgmap v322: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 1.4 KiB/s wr, 0 op/s 2024-02-10T05:54:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:15 smithi192 ceph-mon[52881]: from='mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472]' entity='mds.foofs.smithi028.xhtowe' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.28:0/1798128558"}]': finished 2024-02-10T05:54:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:15 smithi192 ceph-mon[52881]: osdmap e162: 8 total, 8 up, 8 in 2024-02-10T05:54:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:15.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:54:15.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:15 smithi028 ceph-mon[66336]: Evicting (and blocklisting) client session 14542 (172.21.15.28:0/1798128558) 2024-02-10T05:54:15.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:15 smithi028 ceph-mon[66336]: pgmap v322: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 1.4 KiB/s wr, 0 op/s 2024-02-10T05:54:15.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:15 smithi028 ceph-mon[66336]: from='mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472]' entity='mds.foofs.smithi028.xhtowe' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.28:0/1798128558"}]': finished 2024-02-10T05:54:15.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:15 smithi028 ceph-mon[66336]: osdmap e162: 8 total, 8 up, 8 in 2024-02-10T05:54:15.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:15.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:54:16.497 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:54:16.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: Upgrade: Updating nfs.foo.1.0.smithi192.xgmliu (2/2) 2024-02-10T05:54:16.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:16.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: Creating key for client.nfs.foo.1.0.smithi192.xgmliu 2024-02-10T05:54:16.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi192.xgmliu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T05:54:16.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-02-10T05:54:16.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T05:54:16.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T05:54:16.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:16.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T05:54:16.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T05:54:16.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: Rados config object exists: conf-nfs.foo 2024-02-10T05:54:16.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: Creating key for client.nfs.foo.1.0.smithi192.xgmliu-rgw 2024-02-10T05:54:16.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi192.xgmliu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:54:16.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: Bind address in nfs.foo.1.0.smithi192.xgmliu's ganesha conf is defaulting to empty 2024-02-10T05:54:16.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:16.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: Deploying daemon nfs.foo.1.0.smithi192.xgmliu on smithi192 2024-02-10T05:54:16.810 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:16 smithi028 ceph-mon[66336]: pgmap v324: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-02-10T05:54:17.048 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: Upgrade: Updating nfs.foo.1.0.smithi192.xgmliu (2/2) 2024-02-10T05:54:17.048 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:17.049 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: Creating key for client.nfs.foo.1.0.smithi192.xgmliu 2024-02-10T05:54:17.049 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi192.xgmliu", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-10T05:54:17.049 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-02-10T05:54:17.049 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-10T05:54:17.049 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-10T05:54:17.049 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:17.049 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-10T05:54:17.049 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-10T05:54:17.049 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: Rados config object exists: conf-nfs.foo 2024-02-10T05:54:17.049 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: Creating key for client.nfs.foo.1.0.smithi192.xgmliu-rgw 2024-02-10T05:54:17.049 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi192.xgmliu-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:54:17.049 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: Bind address in nfs.foo.1.0.smithi192.xgmliu's ganesha conf is defaulting to empty 2024-02-10T05:54:17.050 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:17.050 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: Deploying daemon nfs.foo.1.0.smithi192.xgmliu on smithi192 2024-02-10T05:54:17.050 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:16 smithi192 ceph-mon[52881]: pgmap v324: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.3 KiB/s wr, 3 op/s 2024-02-10T05:54:17.168 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:54:17.168 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (10m) 34s ago 19m 23.8M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:54:17.168 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (6m) 34s ago 19m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:54:17.168 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (6m) 13s ago 18m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:54:17.168 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (9m) 34s ago 19m 45.9M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (40s) 34s ago 13m 11.1M - 19.0.0-1361-g290212a1 c328ab9dcfc7 91720c6bc17a 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (17s) 13s ago 13m 12.5M - 19.0.0-1361-g290212a1 c328ab9dcfc7 fe0331b6e3e4 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (9m) 34s ago 20m 510M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (8m) 13s ago 18m 454M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (8m) 34s ago 21m 59.5M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (7m) 13s ago 18m 47.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 starting - - - - 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (13m) 13s ago 13m 86.4M - 4.0 e1d6a67b021e 9926a40334aa 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (9m) 34s ago 19m 9801k - 1.3.1 1dbe0e931976 b8a2349460c2 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (9m) 13s ago 18m 17.7M - 1.3.1 1dbe0e931976 9d4423402228 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (5m) 34s ago 18m 406M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (5m) 34s ago 17m 369M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (4m) 34s ago 17m 353M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 a45f0e24ae44 2024-02-10T05:54:17.169 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (3m) 34s ago 16m 393M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 25f4cf94969e 2024-02-10T05:54:17.170 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (2m) 13s ago 16m 341M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 b15b2733ec77 2024-02-10T05:54:17.170 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (2m) 13s ago 15m 313M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 000913de61d6 2024-02-10T05:54:17.170 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (101s) 13s ago 15m 286M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 c9776c8b4f21 2024-02-10T05:54:17.170 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (65s) 13s ago 15m 297M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 86dbef76e22d 2024-02-10T05:54:17.170 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (8m) 34s ago 18m 65.6M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:54:17.573 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:54:17.573 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:54:17.573 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:54:17.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:54:17.573 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:54:17.573 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:54:17.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:54:17.573 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:54:17.573 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T05:54:17.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:54:17.573 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:54:17.573 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:54:17.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:54:17.574 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:54:17.574 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 14 2024-02-10T05:54:17.574 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:54:17.574 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:54:17.899 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:17 smithi192 ceph-mon[52881]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:17.899 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:17 smithi192 ceph-mon[52881]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:17.899 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:17 smithi192 ceph-mon[52881]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:17.899 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:17 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/1780026359' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:17.914 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:54:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:54:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:54:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:54:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:54:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "mds", 2024-02-10T05:54:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:54:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:54:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "osd", 2024-02-10T05:54:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "mon" 2024-02-10T05:54:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:54:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "16/23 daemons upgraded", 2024-02-10T05:54:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading nfs daemons", 2024-02-10T05:54:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:54:17.915 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:54:17.915 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:17 smithi028 ceph-mon[66336]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:17.915 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:17 smithi028 ceph-mon[66336]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:17.915 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:17 smithi028 ceph-mon[66336]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:17.915 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:17 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/1780026359' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:18.322 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:54:19.007 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:18 smithi192 ceph-mon[52881]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:19.007 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:18 smithi192 ceph-mon[52881]: pgmap v325: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T05:54:19.007 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:18 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2234198513' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:54:19.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:18 smithi028 ceph-mon[66336]: from='client.34536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:19.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:18 smithi028 ceph-mon[66336]: pgmap v325: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T05:54:19.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:18 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2234198513' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:54:20.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:20.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:20.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:20 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:20.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:20 smithi028 ceph-mon[66336]: pgmap v326: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T05:54:20.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:20.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:20.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:20 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:20.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:20 smithi192 ceph-mon[52881]: pgmap v326: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T05:54:23.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:23 smithi028 ceph-mon[66336]: pgmap v327: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-10T05:54:23.490 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:23.490 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:23 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:23.490 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:23 smithi028 ceph-mon[66336]: from='mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472]' entity='mds.foofs.smithi028.xhtowe' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.192:0/2291645479"}]: dispatch 2024-02-10T05:54:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:23 smithi192 ceph-mon[52881]: pgmap v327: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-10T05:54:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:23 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:23 smithi192 ceph-mon[52881]: from='mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472]' entity='mds.foofs.smithi028.xhtowe' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.192:0/2291645479"}]: dispatch 2024-02-10T05:54:24.368 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:24 smithi028 ceph-mon[66336]: Evicting (and blocklisting) client session 24339 (172.21.15.192:0/2291645479) 2024-02-10T05:54:24.368 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:24 smithi028 ceph-mon[66336]: from='mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472]' entity='mds.foofs.smithi028.xhtowe' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.192:0/2291645479"}]': finished 2024-02-10T05:54:24.368 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:24 smithi028 ceph-mon[66336]: osdmap e163: 8 total, 8 up, 8 in 2024-02-10T05:54:24.368 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:24 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:24.368 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:24 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:24.459 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:24 smithi192 ceph-mon[52881]: Evicting (and blocklisting) client session 24339 (172.21.15.192:0/2291645479) 2024-02-10T05:54:24.459 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:24 smithi192 ceph-mon[52881]: from='mds.? [v2:172.21.15.28:6834/2698552472,v1:172.21.15.28:6835/2698552472]' entity='mds.foofs.smithi028.xhtowe' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.192:0/2291645479"}]': finished 2024-02-10T05:54:24.459 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:24 smithi192 ceph-mon[52881]: osdmap e163: 8 total, 8 up, 8 in 2024-02-10T05:54:24.459 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:24 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:24.459 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:24 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:25 smithi192 ceph-mon[52881]: pgmap v329: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-10T05:54:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:25.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:25 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:25.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:25 smithi028 ceph-mon[66336]: pgmap v329: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-10T05:54:25.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:25.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:25 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:27.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:26 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:27.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:26 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:27.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:26 smithi028 ceph-mon[66336]: pgmap v330: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-10T05:54:27.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:26 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:27.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:26 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:27.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:26 smithi192 ceph-mon[52881]: pgmap v330: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-10T05:54:28.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:28 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:28.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:28 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:28.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:28 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:28.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:28 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:29.442 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:29 smithi028 ceph-mon[66336]: pgmap v331: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T05:54:29.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:29 smithi192 ceph-mon[52881]: pgmap v331: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T05:54:30.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:54:30.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:54:31.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: pgmap v332: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T05:54:31.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.d"}]: dispatch 2024-02-10T05:54:31.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1c"}]: dispatch 2024-02-10T05:54:31.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:31.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:31.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:31.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:31.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:31.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi028.tzitkz"}]: dispatch 2024-02-10T05:54:31.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi028.tzitkz"}]': finished 2024-02-10T05:54:31.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi192.xgmliu"}]: dispatch 2024-02-10T05:54:31.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi192.xgmliu"}]': finished 2024-02-10T05:54:31.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.561 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: pgmap v332: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T05:54:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.d"}]: dispatch 2024-02-10T05:54:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1c"}]: dispatch 2024-02-10T05:54:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:31.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi028.tzitkz"}]: dispatch 2024-02-10T05:54:31.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi028.tzitkz"}]': finished 2024-02-10T05:54:31.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi192.xgmliu"}]: dispatch 2024-02-10T05:54:31.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi192.xgmliu"}]': finished 2024-02-10T05:54:31.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:31.574 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:32.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:32 smithi028 ceph-mon[66336]: Upgrade: Setting container_image for all nfs 2024-02-10T05:54:32.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:32 smithi028 ceph-mon[66336]: Upgrade: Setting container_image for all nvmeof 2024-02-10T05:54:32.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:32 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.d"}]': finished 2024-02-10T05:54:32.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:32 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1c"}]': finished 2024-02-10T05:54:32.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:32 smithi028 ceph-mon[66336]: osdmap e164: 8 total, 8 up, 8 in 2024-02-10T05:54:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:32 smithi192 ceph-mon[52881]: Upgrade: Setting container_image for all nfs 2024-02-10T05:54:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:32 smithi192 ceph-mon[52881]: Upgrade: Setting container_image for all nvmeof 2024-02-10T05:54:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:32 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.d"}]': finished 2024-02-10T05:54:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:32 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1c"}]': finished 2024-02-10T05:54:32.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:32 smithi192 ceph-mon[52881]: osdmap e164: 8 total, 8 up, 8 in 2024-02-10T05:54:33.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:33 smithi028 ceph-mon[66336]: pgmap v334: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 122 B/s rd, 488 B/s wr, 0 op/s 2024-02-10T05:54:33.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:33 smithi028 ceph-mon[66336]: osdmap e165: 8 total, 8 up, 8 in 2024-02-10T05:54:33.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:33 smithi192 ceph-mon[52881]: pgmap v334: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 122 B/s rd, 488 B/s wr, 0 op/s 2024-02-10T05:54:33.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:33 smithi192 ceph-mon[52881]: osdmap e165: 8 total, 8 up, 8 in 2024-02-10T05:54:34.356 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:34 smithi028 ceph-mon[66336]: Upgrade: Updating node-exporter.smithi028 (1/2) 2024-02-10T05:54:34.356 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:34 smithi028 ceph-mon[66336]: Deploying daemon node-exporter.smithi028 on smithi028 2024-02-10T05:54:34.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:34 smithi192 ceph-mon[52881]: Upgrade: Updating node-exporter.smithi028 (1/2) 2024-02-10T05:54:34.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:34 smithi192 ceph-mon[52881]: Deploying daemon node-exporter.smithi028 on smithi028 2024-02-10T05:54:35.233 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:35 smithi028 ceph-mon[66336]: pgmap v336: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-02-10T05:54:35.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:35 smithi192 ceph-mon[52881]: pgmap v336: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-02-10T05:54:37.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:37 smithi028 ceph-mon[66336]: pgmap v337: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T05:54:37.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:37 smithi192 ceph-mon[52881]: pgmap v337: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T05:54:39.276 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:38 smithi192 ceph-mon[52881]: pgmap v338: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T05:54:39.276 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:38 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:39.276 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:38 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:39.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:38 smithi028 ceph-mon[66336]: pgmap v338: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T05:54:39.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:38 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:39.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:38 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:41.536 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:41 smithi192 ceph-mon[52881]: pgmap v339: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 231 B/s rd, 0 op/s 2024-02-10T05:54:41.536 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:41 smithi192 ceph-mon[52881]: Upgrade: Updating node-exporter.smithi192 (2/2) 2024-02-10T05:54:41.536 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:41 smithi192 ceph-mon[52881]: Deploying daemon node-exporter.smithi192 on smithi192 2024-02-10T05:54:41.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:41 smithi028 ceph-mon[66336]: pgmap v339: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 231 B/s rd, 0 op/s 2024-02-10T05:54:41.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:41 smithi028 ceph-mon[66336]: Upgrade: Updating node-exporter.smithi192 (2/2) 2024-02-10T05:54:41.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:41 smithi028 ceph-mon[66336]: Deploying daemon node-exporter.smithi192 on smithi192 2024-02-10T05:54:43.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:43 smithi028 ceph-mon[66336]: pgmap v340: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-02-10T05:54:43.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:43 smithi192 ceph-mon[52881]: pgmap v340: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-02-10T05:54:45.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:45 smithi192 ceph-mon[52881]: pgmap v341: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2024-02-10T05:54:45.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:54:45.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:45 smithi028 ceph-mon[66336]: pgmap v341: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 172 B/s rd, 0 op/s 2024-02-10T05:54:45.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:54:46.551 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:46.551 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:46.551 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:46.551 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:46 smithi192 ceph-mon[52881]: pgmap v342: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:54:46.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:46.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:46.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:46.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:46 smithi028 ceph-mon[66336]: pgmap v342: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:54:48.706 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:54:49.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:48 smithi028 ceph-mon[66336]: pgmap v343: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:54:49.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:49.059 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:48 smithi192 ceph-mon[52881]: pgmap v343: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:54:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:49.406 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (10m) 24s ago 20m 23.9M - 0.23.0 ba2b418f427c e4e5c0e45cad 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (7m) 24s ago 20m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (6m) 0s ago 19m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (10m) 24s ago 19m 51.9M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (72s) 24s ago 14m 33.5M - 19.0.0-1361-g290212a1 c328ab9dcfc7 91720c6bc17a 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (49s) 0s ago 14m 13.9M - 19.0.0-1361-g290212a1 c328ab9dcfc7 fe0331b6e3e4 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (9m) 24s ago 21m 511M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (8m) 0s ago 19m 455M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (8m) 24s ago 21m 61.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (8m) 0s ago 19m 47.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (35s) 24s ago 13m 69.3M - 5.7 c328ab9dcfc7 144af8b0ce0f 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (30s) 0s ago 13m 68.8M - 5.7 c328ab9dcfc7 4da7771981c8 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 starting - - - - 2024-02-10T05:54:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (4s) 0s ago 19m 5444k - 1.5.0 0da6a335fe13 68b6dd71219e 2024-02-10T05:54:49.408 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (6m) 24s ago 18m 408M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:54:49.408 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (5m) 24s ago 18m 371M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:54:49.408 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (4m) 24s ago 17m 354M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 a45f0e24ae44 2024-02-10T05:54:49.408 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (3m) 24s ago 17m 395M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 25f4cf94969e 2024-02-10T05:54:49.408 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (3m) 0s ago 16m 342M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 b15b2733ec77 2024-02-10T05:54:49.408 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (2m) 0s ago 16m 313M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 000913de61d6 2024-02-10T05:54:49.408 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (2m) 0s ago 16m 286M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 c9776c8b4f21 2024-02-10T05:54:49.408 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (97s) 0s ago 15m 296M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 86dbef76e22d 2024-02-10T05:54:49.408 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (9m) 24s ago 19m 65.7M - 2.33.4 514e6a882f6e b4bdd2bbbdee 2024-02-10T05:54:49.824 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:54:49.824 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:54:49.824 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:54:49.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:54:49.824 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:54:49.825 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:54:49.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:54:49.825 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:54:49.825 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T05:54:49.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:54:49.825 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:54:49.825 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:54:49.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:54:49.825 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:54:49.825 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 14 2024-02-10T05:54:49.825 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:54:49.825 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:54:50.204 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:49 smithi028 ceph-mon[66336]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:50.204 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:49 smithi028 ceph-mon[66336]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:50.204 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:49 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:50.204 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:49 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:50.204 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:49 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/1607106588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout: "mds", 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout: "osd", 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout: "mon", 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs" 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:54:50.205 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "18/23 daemons upgraded", 2024-02-10T05:54:50.206 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading node-exporter daemons", 2024-02-10T05:54:50.206 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:54:50.206 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:54:50.305 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:49 smithi192 ceph-mon[52881]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:50.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:49 smithi192 ceph-mon[52881]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:50.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:49 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:50.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:49 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:50.306 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:49 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/1607106588' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:50.609 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:54:51.288 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:50 smithi192 ceph-mon[52881]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:51.288 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:50 smithi192 ceph-mon[52881]: pgmap v344: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:54:51.288 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:50 smithi192 ceph-mon[52881]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:51.288 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:50 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2735875041' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:54:51.289 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:50 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:51.289 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:50 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:51.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:50 smithi028 ceph-mon[66336]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:51.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:50 smithi028 ceph-mon[66336]: pgmap v344: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:54:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:50 smithi028 ceph-mon[66336]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:50 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2735875041' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:54:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:50 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:51.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:50 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:53.028 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:52 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:53.028 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:52 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:53.028 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:52 smithi192 ceph-mon[52881]: pgmap v345: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:54:53.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:52 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:53.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:52 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:53.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:52 smithi028 ceph-mon[66336]: pgmap v345: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:54:54.356 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:54 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:54.357 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:54 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:54.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:54 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:54.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:54 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:55.272 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:55 smithi028 ceph-mon[66336]: pgmap v346: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:54:55.472 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:55 smithi192 ceph-mon[52881]: pgmap v346: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:54:56.439 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:56.439 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:56.439 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:56.439 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:56.439 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:56.439 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:56.439 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:56.439 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.439 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.440 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.440 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.440 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.440 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.440 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.440 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.440 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.440 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.440 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.440 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.440 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:56 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.747 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:56.747 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:56.747 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:56.747 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:56.747 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:56.747 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:54:56.747 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:56.747 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.747 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.747 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.747 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.747 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.748 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.748 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.748 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.748 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.748 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.748 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.748 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:56.748 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:56 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:54:57.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:57 smithi028 ceph-mon[66336]: pgmap v347: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:54:57.821 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:57 smithi192 ceph-mon[52881]: pgmap v347: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:54:58.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:58 smithi028 ceph-mon[66336]: Upgrade: Updating prometheus.smithi028 2024-02-10T05:54:58.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:58 smithi028 ceph-mon[66336]: Deploying daemon prometheus.smithi028 on smithi028 2024-02-10T05:54:58.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:58 smithi192 ceph-mon[52881]: Upgrade: Updating prometheus.smithi028 2024-02-10T05:54:58.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:58 smithi192 ceph-mon[52881]: Deploying daemon prometheus.smithi028 on smithi028 2024-02-10T05:54:59.547 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:54:59 smithi028 ceph-mon[66336]: pgmap v348: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:54:59.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:54:59 smithi192 ceph-mon[52881]: pgmap v348: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:00.730 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:00.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:01.385 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:01 smithi028 ceph-mon[66336]: pgmap v349: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:01.823 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:01 smithi192 ceph-mon[52881]: pgmap v349: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:02.727 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:02 smithi028 ceph-mon[66336]: pgmap v350: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:02.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:02 smithi192 ceph-mon[52881]: pgmap v350: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:04.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:03 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:04.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:03 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:04.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:03 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:04.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:03 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:04.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:03 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:04.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:03 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:04.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:03 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:04.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:03 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:04.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:03 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:04.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:03 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:05.294 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:04 smithi028 ceph-mon[66336]: pgmap v351: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:05.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:04 smithi192 ceph-mon[52881]: pgmap v351: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:07.371 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:07 smithi028 ceph-mon[66336]: pgmap v352: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:55:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:07 smithi192 ceph-mon[52881]: pgmap v352: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:55:09.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:08 smithi028 ceph-mon[66336]: pgmap v353: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:09.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:09.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:08 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:08 smithi192 ceph-mon[52881]: pgmap v353: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:09.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:08 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:10.773 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:10.773 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:10 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:10.773 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:10 smithi028 ceph-mon[66336]: pgmap v354: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:10.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:10.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:10 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:10.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:10 smithi192 ceph-mon[52881]: pgmap v354: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: pgmap v355: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:55:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:13.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.560 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:13 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: pgmap v355: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:13.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:13.573 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:13 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:14.288 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:14 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:55:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:14 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:55:15.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:15 smithi028 ceph-mon[66336]: pgmap v356: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:15.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:15.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:15 smithi192 ceph-mon[52881]: pgmap v356: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:16.463 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:16 smithi028 ceph-mon[66336]: Upgrade: Updating alertmanager.smithi028 2024-02-10T05:55:16.464 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:16 smithi028 ceph-mon[66336]: Deploying daemon alertmanager.smithi028 on smithi028 2024-02-10T05:55:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:16 smithi192 ceph-mon[52881]: Upgrade: Updating alertmanager.smithi028 2024-02-10T05:55:16.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:16 smithi192 ceph-mon[52881]: Deploying daemon alertmanager.smithi028 on smithi028 2024-02-10T05:55:17.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:17 smithi028 ceph-mon[66336]: pgmap v357: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:55:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:17 smithi192 ceph-mon[52881]: pgmap v357: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:55:19.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:18 smithi028 ceph-mon[66336]: pgmap v358: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:19.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:18 smithi192 ceph-mon[52881]: pgmap v358: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:20.985 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:55:21.271 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:21 smithi028 ceph-mon[66336]: pgmap v359: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:21.271 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:21 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:21.271 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:21 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:21.271 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:21 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:21.271 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:21 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:21.271 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:21 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:21.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:21 smithi192 ceph-mon[52881]: pgmap v359: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:21.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:21 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:21.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:21 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:21.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:21 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:21.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:21 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:21.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:21 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:21.672 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:55:21.672 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 starting - - - - 2024-02-10T05:55:21.672 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (7m) 13s ago 20m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (7m) 33s ago 20m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (11m) 13s ago 20m 51.8M - 8.3.5 dad864ee21e9 bfcd32c0968e 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (104s) 13s ago 14m 33.9M - 19.0.0-1361-g290212a1 c328ab9dcfc7 91720c6bc17a 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (81s) 33s ago 14m 13.9M - 19.0.0-1361-g290212a1 c328ab9dcfc7 fe0331b6e3e4 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (10m) 13s ago 21m 514M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (9m) 33s ago 20m 455M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (9m) 13s ago 22m 62.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (8m) 33s ago 19m 47.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (68s) 13s ago 14m 69.5M - 5.7 c328ab9dcfc7 144af8b0ce0f 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (62s) 33s ago 14m 68.8M - 5.7 c328ab9dcfc7 4da7771981c8 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (43s) 13s ago 20m 7852k - 1.5.0 0da6a335fe13 9fa372730507 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (36s) 33s ago 19m 5444k - 1.5.0 0da6a335fe13 68b6dd71219e 2024-02-10T05:55:21.673 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (6m) 13s ago 19m 409M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:55:21.674 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (6m) 13s ago 18m 372M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:55:21.674 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (5m) 13s ago 18m 356M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 a45f0e24ae44 2024-02-10T05:55:21.674 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (4m) 13s ago 17m 397M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 25f4cf94969e 2024-02-10T05:55:21.674 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (4m) 33s ago 17m 342M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 b15b2733ec77 2024-02-10T05:55:21.674 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (3m) 33s ago 17m 313M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 000913de61d6 2024-02-10T05:55:21.674 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (2m) 33s ago 16m 286M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 c9776c8b4f21 2024-02-10T05:55:21.674 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (2m) 33s ago 16m 296M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 86dbef76e22d 2024-02-10T05:55:21.674 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (19s) 13s ago 19m 42.4M - 2.43.0 a07b618ecd1d aca95a097e67 2024-02-10T05:55:22.100 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:55:22.101 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:55:22.102 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 14 2024-02-10T05:55:22.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:55:22.102 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:55:22.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:22 smithi028 ceph-mon[66336]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:22.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:22 smithi028 ceph-mon[66336]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:22.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:22 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/4223970052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:22.442 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:55:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T05:55:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": true, 2024-02-10T05:55:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-10T05:55:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [ 2024-02-10T05:55:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "mds", 2024-02-10T05:55:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-02-10T05:55:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr", 2024-02-10T05:55:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "osd", 2024-02-10T05:55:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "mon", 2024-02-10T05:55:22.442 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs" 2024-02-10T05:55:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:55:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": "18/23 daemons upgraded", 2024-02-10T05:55:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "Currently upgrading alertmanager daemons", 2024-02-10T05:55:22.443 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:55:22.443 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:55:22.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:22 smithi192 ceph-mon[52881]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:22.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:22 smithi192 ceph-mon[52881]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:22.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:22 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/4223970052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:22.882 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:55:23.461 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:23 smithi028 ceph-mon[66336]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:23.461 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:23 smithi028 ceph-mon[66336]: pgmap v360: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:23.462 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:23 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2078317208' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:55:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:23 smithi192 ceph-mon[52881]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:23 smithi192 ceph-mon[52881]: pgmap v360: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:23 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2078317208' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:55:24.442 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:24 smithi028 ceph-mon[66336]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:24.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:24 smithi192 ceph-mon[52881]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:25.473 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:25 smithi192 ceph-mon[52881]: pgmap v361: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:25.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:25 smithi028 ceph-mon[66336]: pgmap v361: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:26.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:26 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:26.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:26 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:26.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:26 smithi028 ceph-mon[66336]: pgmap v362: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:55:26.821 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:26 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:26.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:26 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:26.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:26 smithi192 ceph-mon[52881]: pgmap v362: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:55:27.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:27 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:27.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:27 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:27 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:28.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:27 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:29.180 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:28 smithi028 ceph-mon[66336]: pgmap v363: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:28 smithi192 ceph-mon[52881]: pgmap v363: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:31.038 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:31.038 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:31.038 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:31.038 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:31.038 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:31.038 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: pgmap v364: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.039 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.040 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.040 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.040 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.040 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.040 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:31 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: pgmap v364: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:31.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:31 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:33.184 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:33 smithi028 ceph-mon[66336]: Upgrade: Updating grafana.smithi028 2024-02-10T05:55:33.184 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:33 smithi028 ceph-mon[66336]: Deploying daemon grafana.smithi028 on smithi028 2024-02-10T05:55:33.184 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:33 smithi028 ceph-mon[66336]: pgmap v365: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:33.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:33 smithi192 ceph-mon[52881]: Upgrade: Updating grafana.smithi028 2024-02-10T05:55:33.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:33 smithi192 ceph-mon[52881]: Deploying daemon grafana.smithi028 on smithi028 2024-02-10T05:55:33.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:33 smithi192 ceph-mon[52881]: pgmap v365: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:35.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:35 smithi028 ceph-mon[66336]: pgmap v366: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:35.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:35 smithi192 ceph-mon[52881]: pgmap v366: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:37.403 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:37 smithi028 ceph-mon[66336]: pgmap v367: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:55:37.404 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:37 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:37.404 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:37 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:37.404 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:37 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:37 smithi192 ceph-mon[52881]: pgmap v367: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:55:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:37 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:37 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:37 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:39.204 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:38 smithi028 ceph-mon[66336]: pgmap v368: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:39.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:38 smithi192 ceph-mon[52881]: pgmap v368: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:41.405 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:41 smithi028 ceph-mon[66336]: pgmap v369: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:41 smithi192 ceph-mon[52881]: pgmap v369: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:43.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:42 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:43.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:42 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:43.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:42 smithi028 ceph-mon[66336]: pgmap v370: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:43.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:42 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:42 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:43.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:42 smithi192 ceph-mon[52881]: pgmap v370: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:44.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:44.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:43 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:44.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:44.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:43 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:45.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:44 smithi028 ceph-mon[66336]: pgmap v371: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:45.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:44 smithi192 ceph-mon[52881]: pgmap v371: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:46.058 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:46.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:47.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: pgmap v372: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:55:47.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.310 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T05:55:47.311 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.312 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.313 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T05:55:47.314 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T05:55:47.314 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:47.314 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:47.314 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:47.314 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.314 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.314 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:47.314 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:47.314 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:47.314 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.314 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:46 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: pgmap v372: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:55:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:47.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.323 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-10T05:55:47.324 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-10T05:55:47.325 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-10T05:55:47.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-10T05:55:47.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-10T05:55:47.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-10T05:55:47.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-10T05:55:47.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.326 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:47.327 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:46 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:48.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:47 smithi028 ceph-mon[66336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:55:48.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:47 smithi028 ceph-mon[66336]: Upgrade: Finalizing container_image settings 2024-02-10T05:55:48.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:47 smithi028 ceph-mon[66336]: Upgrade: Complete! 2024-02-10T05:55:48.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:47 smithi192 ceph-mon[52881]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:55:48.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:47 smithi192 ceph-mon[52881]: Upgrade: Finalizing container_image settings 2024-02-10T05:55:48.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:47 smithi192 ceph-mon[52881]: Upgrade: Complete! 2024-02-10T05:55:49.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:48 smithi028 ceph-mon[66336]: pgmap v373: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:49.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:48 smithi192 ceph-mon[52881]: pgmap v373: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:51.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:51 smithi028 ceph-mon[66336]: pgmap v374: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:51.809 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:51 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:51 smithi192 ceph-mon[52881]: pgmap v374: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:51.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:51 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:55:53.535 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:53 smithi028 ceph-mon[66336]: pgmap v375: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:53.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:53 smithi192 ceph-mon[52881]: pgmap v375: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:53.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T05:55:54.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:54 smithi028 ceph-mon[66336]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:54.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:54 smithi192 ceph-mon[52881]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:55.461 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:55 smithi028 ceph-mon[66336]: pgmap v376: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:55.723 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:55 smithi192 ceph-mon[52881]: pgmap v376: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:55.874 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:55:55.874 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (36s) 14s ago 21m 17.8M - 0.25.0 c8568f914cd2 299636b64325 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (8m) 14s ago 21m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (8m) 67s ago 20m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (19s) 14s ago 21m 58.8M - 9.4.12 a0ea5e3c46d9 39bd5f8dc2a4 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (2m) 14s ago 15m 34.1M - 19.0.0-1361-g290212a1 c328ab9dcfc7 91720c6bc17a 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (116s) 67s ago 15m 13.9M - 19.0.0-1361-g290212a1 c328ab9dcfc7 fe0331b6e3e4 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (10m) 14s ago 22m 515M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (10m) 67s ago 20m 455M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (9m) 14s ago 22m 63.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (9m) 67s ago 20m 47.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (102s) 14s ago 15m 70.4M - 5.7 c328ab9dcfc7 144af8b0ce0f 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (96s) 67s ago 14m 68.8M - 5.7 c328ab9dcfc7 4da7771981c8 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (77s) 14s ago 20m 8082k - 1.5.0 0da6a335fe13 9fa372730507 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (70s) 67s ago 20m 5444k - 1.5.0 0da6a335fe13 68b6dd71219e 2024-02-10T05:55:55.875 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (7m) 14s ago 19m 409M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:55:55.876 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (6m) 14s ago 19m 372M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:55:55.876 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (5m) 14s ago 18m 356M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 a45f0e24ae44 2024-02-10T05:55:55.876 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (5m) 14s ago 18m 397M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 25f4cf94969e 2024-02-10T05:55:55.876 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (4m) 67s ago 18m 342M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 b15b2733ec77 2024-02-10T05:55:55.876 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (3m) 67s ago 17m 313M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 000913de61d6 2024-02-10T05:55:55.876 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (3m) 67s ago 17m 286M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 c9776c8b4f21 2024-02-10T05:55:55.876 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (2m) 67s ago 16m 296M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 86dbef76e22d 2024-02-10T05:55:55.876 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (53s) 14s ago 20m 51.5M - 2.43.0 a07b618ecd1d aca95a097e67 2024-02-10T05:55:56.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T05:55:57.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:57 smithi028 ceph-mon[66336]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:57.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:57 smithi028 ceph-mon[66336]: pgmap v377: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:55:57.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:57 smithi192 ceph-mon[52881]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:57.822 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:57 smithi192 ceph-mon[52881]: pgmap v377: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:55:58.727 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:55:58.727 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:55:58.727 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:55:58.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:55:58.727 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:55:58.727 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:55:58.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:55:58.727 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:55:58.728 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T05:55:58.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:55:58.728 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:55:58.728 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:55:58.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:55:58.728 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:55:58.728 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 14 2024-02-10T05:55:58.728 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:55:58.728 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:55:59.022 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:58 smithi028 ceph-mon[66336]: pgmap v378: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:59.022 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:55:58 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2360305725' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:59.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:58 smithi192 ceph-mon[52881]: pgmap v378: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:55:59.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:55:58 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2360305725' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:55:59.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-02-10T05:56:00.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:00.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:01.082 INFO:teuthology.orchestra.run.smithi028.stdout:wait for servicemap items w/ changing names to refresh 2024-02-10T05:56:01.462 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:01 smithi028 ceph-mon[66336]: pgmap v379: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:01.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:01 smithi192 ceph-mon[52881]: pgmap v379: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:01.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'sleep 60' 2024-02-10T05:56:03.438 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:03 smithi028 ceph-mon[66336]: pgmap v380: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:03.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:03 smithi192 ceph-mon[52881]: pgmap v380: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:05.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:05 smithi028 ceph-mon[66336]: pgmap v381: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:05.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:05 smithi192 ceph-mon[52881]: pgmap v381: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:07.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:07 smithi028 ceph-mon[66336]: pgmap v382: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:56:07.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:07 smithi192 ceph-mon[52881]: pgmap v382: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:56:09.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:08 smithi028 ceph-mon[66336]: pgmap v383: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:09.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:08 smithi192 ceph-mon[52881]: pgmap v383: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:11.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:11 smithi028 ceph-mon[66336]: pgmap v384: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:11.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:11 smithi192 ceph-mon[52881]: pgmap v384: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:13.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:13 smithi028 ceph-mon[66336]: pgmap v385: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:13.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:13 smithi192 ceph-mon[52881]: pgmap v385: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:15.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:15 smithi028 ceph-mon[66336]: pgmap v386: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:15.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:15 smithi192 ceph-mon[52881]: pgmap v386: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:17.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:17 smithi028 ceph-mon[66336]: pgmap v387: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:56:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:17 smithi192 ceph-mon[52881]: pgmap v387: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:56:19.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:18 smithi028 ceph-mon[66336]: pgmap v388: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:19.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:18 smithi192 ceph-mon[52881]: pgmap v388: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:21.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:21 smithi028 ceph-mon[66336]: pgmap v389: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:21.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:21 smithi192 ceph-mon[52881]: pgmap v389: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:23.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:23 smithi028 ceph-mon[66336]: pgmap v390: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:23 smithi192 ceph-mon[52881]: pgmap v390: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:25.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:25 smithi028 ceph-mon[66336]: pgmap v391: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:25.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:25 smithi192 ceph-mon[52881]: pgmap v391: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:27.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:27 smithi028 ceph-mon[66336]: pgmap v392: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:56:27.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:27 smithi192 ceph-mon[52881]: pgmap v392: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:56:29.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:28 smithi028 ceph-mon[66336]: pgmap v393: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:28 smithi192 ceph-mon[52881]: pgmap v393: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:30.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:30.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:31.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:31 smithi028 ceph-mon[66336]: pgmap v394: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:31.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:31 smithi192 ceph-mon[52881]: pgmap v394: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:33.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:33 smithi028 ceph-mon[66336]: pgmap v395: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:33.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:33 smithi192 ceph-mon[52881]: pgmap v395: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:35.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:35 smithi028 ceph-mon[66336]: pgmap v396: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:35.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:35 smithi192 ceph-mon[52881]: pgmap v396: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:37.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:37 smithi028 ceph-mon[66336]: pgmap v397: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:56:37.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:37 smithi192 ceph-mon[52881]: pgmap v397: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:56:39.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:38 smithi028 ceph-mon[66336]: pgmap v398: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:39.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:38 smithi192 ceph-mon[52881]: pgmap v398: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:41.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:41 smithi028 ceph-mon[66336]: pgmap v399: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:41.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:41 smithi192 ceph-mon[52881]: pgmap v399: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:43.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:43 smithi028 ceph-mon[66336]: pgmap v400: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:43.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:43 smithi192 ceph-mon[52881]: pgmap v400: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:45.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:45 smithi028 ceph-mon[66336]: pgmap v401: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:45.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:45 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:45.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:45 smithi192 ceph-mon[52881]: pgmap v401: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:45.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:45 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:47.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:47 smithi028 ceph-mon[66336]: pgmap v402: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:56:47.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:47 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:47 smithi192 ceph-mon[52881]: pgmap v402: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:56:47.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:47 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:48.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:48.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:48.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:56:48.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:48 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:56:48.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:48.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:48.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:56:48.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:48 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' 2024-02-10T05:56:49.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:49 smithi028 ceph-mon[66336]: pgmap v403: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:49.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:49 smithi192 ceph-mon[52881]: pgmap v403: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:51.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:51 smithi028 ceph-mon[66336]: pgmap v404: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:51.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:51 smithi192 ceph-mon[52881]: pgmap v404: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:53.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:53 smithi028 ceph-mon[66336]: pgmap v405: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:53.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:53 smithi192 ceph-mon[52881]: pgmap v405: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:55.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:55 smithi028 ceph-mon[66336]: pgmap v406: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:55.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:55 smithi192 ceph-mon[52881]: pgmap v406: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:57.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:57 smithi028 ceph-mon[66336]: pgmap v407: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:56:57.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:57 smithi192 ceph-mon[52881]: pgmap v407: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:56:59.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:56:58 smithi028 ceph-mon[66336]: pgmap v408: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:56:59.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:56:58 smithi192 ceph-mon[52881]: pgmap v408: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:00.559 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:00 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:57:00.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:00 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:57:01.558 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:01 smithi028 ceph-mon[66336]: pgmap v409: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:01.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:01 smithi192 ceph-mon[52881]: pgmap v409: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:03.445 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:03 smithi028 ceph-mon[66336]: pgmap v410: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:03.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:03 smithi192 ceph-mon[52881]: pgmap v410: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:04.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch ps' 2024-02-10T05:57:05.413 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:05 smithi028 ceph-mon[66336]: pgmap v411: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:05.571 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:05 smithi192 ceph-mon[52881]: pgmap v411: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:06.118 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:57:06.118 INFO:teuthology.orchestra.run.smithi028.stdout:alertmanager.smithi028 smithi028 *:9093,9094 running (106s) 84s ago 22m 17.8M - 0.25.0 c8568f914cd2 299636b64325 2024-02-10T05:57:06.118 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi028 smithi028 running (9m) 84s ago 22m 6660k - 19.0.0-1361-g290212a1 c328ab9dcfc7 91c7cb199efe 2024-02-10T05:57:06.118 INFO:teuthology.orchestra.run.smithi028.stdout:crash.smithi192 smithi192 running (9m) 2m ago 21m 6656k - 19.0.0-1361-g290212a1 c328ab9dcfc7 3a4ddc9e23a1 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:grafana.smithi028 smithi028 *:3000 running (90s) 84s ago 22m 58.8M - 9.4.12 a0ea5e3c46d9 39bd5f8dc2a4 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi028.xhtowe smithi028 running (3m) 84s ago 16m 34.1M - 19.0.0-1361-g290212a1 c328ab9dcfc7 91720c6bc17a 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:mds.foofs.smithi192.hdcbzw smithi192 running (3m) 2m ago 16m 13.9M - 19.0.0-1361-g290212a1 c328ab9dcfc7 fe0331b6e3e4 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi028.ucuqfx smithi028 *:8443,9283,8765 running (12m) 84s ago 23m 515M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ae3eb8a6ce02 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.smithi192.hppzcm smithi192 *:8443,9283,8765 running (11m) 2m ago 21m 455M - 19.0.0-1361-g290212a1 c328ab9dcfc7 7545d462e47f 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi028 smithi028 running (10m) 84s ago 23m 63.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57f180a5eb09 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:mon.smithi192 smithi192 running (10m) 2m ago 21m 47.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 89a15b946f09 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.0.0.smithi028.tzitkz smithi028 *:2049 running (2m) 84s ago 16m 70.4M - 5.7 c328ab9dcfc7 144af8b0ce0f 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:nfs.foo.1.0.smithi192.xgmliu smithi192 *:2049 running (2m) 2m ago 16m 68.8M - 5.7 c328ab9dcfc7 4da7771981c8 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi028 smithi028 *:9100 running (2m) 84s ago 22m 8082k - 1.5.0 0da6a335fe13 9fa372730507 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:node-exporter.smithi192 smithi192 *:9100 running (2m) 2m ago 21m 5444k - 1.5.0 0da6a335fe13 68b6dd71219e 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (8m) 84s ago 20m 409M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 13cfdd85a4ed 2024-02-10T05:57:06.119 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi028 running (7m) 84s ago 20m 372M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 84c53ce02608 2024-02-10T05:57:06.120 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi028 running (7m) 84s ago 19m 356M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 a45f0e24ae44 2024-02-10T05:57:06.120 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (6m) 84s ago 19m 397M 1937M 19.0.0-1361-g290212a1 c328ab9dcfc7 25f4cf94969e 2024-02-10T05:57:06.120 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi192 running (5m) 2m ago 19m 342M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 b15b2733ec77 2024-02-10T05:57:06.120 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi192 running (5m) 2m ago 18m 313M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 000913de61d6 2024-02-10T05:57:06.120 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi192 running (4m) 2m ago 18m 286M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 c9776c8b4f21 2024-02-10T05:57:06.120 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi192 running (3m) 2m ago 17m 296M 2704M 19.0.0-1361-g290212a1 c328ab9dcfc7 86dbef76e22d 2024-02-10T05:57:06.120 INFO:teuthology.orchestra.run.smithi028.stdout:prometheus.smithi028 smithi028 *:9095 running (2m) 84s ago 21m 51.5M - 2.43.0 a07b618ecd1d aca95a097e67 2024-02-10T05:57:06.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph orch upgrade status' 2024-02-10T05:57:07.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:07 smithi028 ceph-mon[66336]: pgmap v412: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:57:07.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:07 smithi028 ceph-mon[66336]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:07 smithi192 ceph-mon[52881]: pgmap v412: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:57:07.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:07 smithi192 ceph-mon[52881]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:08.686 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:57:08.687 INFO:teuthology.orchestra.run.smithi028.stdout: "target_image": null, 2024-02-10T05:57:08.687 INFO:teuthology.orchestra.run.smithi028.stdout: "in_progress": false, 2024-02-10T05:57:08.687 INFO:teuthology.orchestra.run.smithi028.stdout: "which": "", 2024-02-10T05:57:08.687 INFO:teuthology.orchestra.run.smithi028.stdout: "services_complete": [], 2024-02-10T05:57:08.687 INFO:teuthology.orchestra.run.smithi028.stdout: "progress": null, 2024-02-10T05:57:08.687 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-02-10T05:57:08.687 INFO:teuthology.orchestra.run.smithi028.stdout: "is_paused": false 2024-02-10T05:57:08.687 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:57:09.291 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:08 smithi028 ceph-mon[66336]: pgmap v413: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:09.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph health detail' 2024-02-10T05:57:09.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:08 smithi192 ceph-mon[52881]: pgmap v413: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:10.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:09 smithi028 ceph-mon[66336]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:10.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:09 smithi192 ceph-mon[52881]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:11.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:10 smithi028 ceph-mon[66336]: pgmap v414: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:11.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:10 smithi192 ceph-mon[52881]: pgmap v414: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:11.441 INFO:teuthology.orchestra.run.smithi028.stdout:HEALTH_OK 2024-02-10T05:57:12.030 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:11 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2994476025' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:57:12.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions' 2024-02-10T05:57:12.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:11 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2994476025' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-10T05:57:13.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:12 smithi028 ceph-mon[66336]: pgmap v415: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:13.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:12 smithi192 ceph-mon[52881]: pgmap v415: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:14.126 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-10T05:57:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": { 2024-02-10T05:57:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:57:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:57:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": { 2024-02-10T05:57:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:57:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:57:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": { 2024-02-10T05:57:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 8 2024-02-10T05:57:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:57:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-02-10T05:57:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 2 2024-02-10T05:57:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:57:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "overall": { 2024-02-10T05:57:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)": 14 2024-02-10T05:57:14.127 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:57:14.127 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-10T05:57:14.445 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:14 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/4192197604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:57:14.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:14 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/4192197604' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:57:14.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-02-10T05:57:15.415 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:15 smithi028 ceph-mon[66336]: pgmap v416: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:15.416 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:15 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:57:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:15 smithi192 ceph-mon[52881]: pgmap v416: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:15.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:15 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:57:16.888 INFO:teuthology.orchestra.run.smithi028.stdout:true 2024-02-10T05:57:17.459 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:17 smithi028 ceph-mon[66336]: pgmap v417: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:57:17.460 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:17 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/1361935967' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:57:17.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -e sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-02-10T05:57:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:17 smithi192 ceph-mon[52881]: pgmap v417: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:57:17.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:17 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/1361935967' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:57:18.978 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:18 smithi028 ceph-mon[66336]: pgmap v418: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:19.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:18 smithi192 ceph-mon[52881]: pgmap v418: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:19.518 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)" 2024-02-10T05:57:20.098 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:19 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2077814501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:57:20.125 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-02-10T05:57:20.125 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-02-10T05:57:20.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- ceph orch ls -f json 2024-02-10T05:57:20.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:19 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2077814501' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-10T05:57:20.997 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:20 smithi028 ceph-mon[66336]: pgmap v419: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:21.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:20 smithi192 ceph-mon[52881]: pgmap v419: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:22.070 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-10T05:57:22.070 INFO:teuthology.orchestra.run.smithi028.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-10T05:34:02.862424Z", "last_refresh": "2024-02-10T05:55:41.862643Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-10T05:33:56.744993Z", "last_refresh": "2024-02-10T05:54:48.654867Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-02-10T05:33:59.400736Z", "last_refresh": "2024-02-10T05:55:41.862805Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-10T05:40:28.946230Z", "last_refresh": "2024-02-10T05:54:48.655518Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-10T05:33:54.804111Z", "last_refresh": "2024-02-10T05:54:48.654969Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi028:172.21.15.28=smithi028", "smithi192:172.21.15.192=smithi192"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-10T05:34:45.473096Z", "last_refresh": "2024-02-10T05:54:48.655062Z", "running": 2, "size": 2}}, {"events": ["2024-02-10T05:56:47.642245Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-02-10T05:40:49.248807Z", "last_refresh": "2024-02-10T05:54:48.655593Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-10T05:34:01.255398Z", "last_refresh": "2024-02-10T05:54:48.655140Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", "last_refresh": "2024-02-10T05:54:48.655215Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-10T05:33:57.959700Z", "last_refresh": "2024-02-10T05:55:41.862964Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-10T05:57:22.681 INFO:tasks.cephadm:nfs.foo has 2/2 2024-02-10T05:57:22.681 DEBUG:teuthology.parallel:result is None 2024-02-10T05:57:22.681 INFO:teuthology.run_tasks:Running task vip.exec... 2024-02-10T05:57:22.691 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T05:57:22.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-02-10T05:57:22.721 INFO:teuthology.orchestra.run.smithi028.stderr:+ umount /mnt/foo 2024-02-10T05:57:22.732 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T05:57:22.741 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T05:57:22.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-02-10T05:57:23.382 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:23 smithi028 ceph-mon[66336]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:57:23.382 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:23 smithi028 ceph-mon[66336]: pgmap v420: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:23 smithi192 ceph-mon[52881]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:57:23.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:23 smithi192 ceph-mon[52881]: pgmap v420: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "foo" 2024-02-10T05:57:25.424 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:25 smithi028 ceph-mon[66336]: pgmap v421: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:25.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'ceph nfs export ls foo --detailed' 2024-02-10T05:57:25.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:25 smithi192 ceph-mon[52881]: pgmap v421: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:26.424 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:26 smithi028 ceph-mon[66336]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:26.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:26 smithi192 ceph-mon[52881]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:27.467 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-10T05:57:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-10T05:57:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "access_type": "RW", 2024-02-10T05:57:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "clients": [], 2024-02-10T05:57:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_id": "foo", 2024-02-10T05:57:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "export_id": 1, 2024-02-10T05:57:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "fsal": { 2024-02-10T05:57:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "foofs", 2024-02-10T05:57:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "CEPH", 2024-02-10T05:57:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: "user_id": "nfs.foo.1" 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "protocols": [ 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: 4 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "pseudo": "/fake", 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "security_label": true, 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "squash": "none", 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "transports": [ 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: "TCP" 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-10T05:57:27.468 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-10T05:57:27.469 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:27 smithi028 ceph-mon[66336]: pgmap v422: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:57:27.572 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:27 smithi192 ceph-mon[52881]: pgmap v422: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T05:57:28.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'rados -p .nfs --all ls -' 2024-02-10T05:57:29.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:28 smithi028 ceph-mon[66336]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:29.309 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:28 smithi028 ceph-mon[66336]: pgmap v423: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:29.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:28 smithi192 ceph-mon[52881]: from='client.34646 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:29.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:28 smithi192 ceph-mon[52881]: pgmap v423: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:29.767 INFO:teuthology.orchestra.run.smithi028.stdout:foo rec-0000000000000004:nfs.foo.1 2024-02-10T05:57:29.767 INFO:teuthology.orchestra.run.smithi028.stdout:foo grace 2024-02-10T05:57:29.767 INFO:teuthology.orchestra.run.smithi028.stdout:foo rec-0000000000000004:nfs.foo.0 2024-02-10T05:57:29.767 INFO:teuthology.orchestra.run.smithi028.stdout:foo conf-nfs.foo 2024-02-10T05:57:29.767 INFO:teuthology.orchestra.run.smithi028.stdout:foo export-1 2024-02-10T05:57:30.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:29 smithi028 ceph-mon[66336]: mgrmap e44: smithi028.ucuqfx(active, since 10m), standbys: smithi192.hppzcm 2024-02-10T05:57:30.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:29 smithi192 ceph-mon[52881]: mgrmap e44: smithi028.ucuqfx(active, since 10m), standbys: smithi192.hppzcm 2024-02-10T05:57:30.350 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T05:57:30.361 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T05:57:30.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 -- bash -c 'set -ex 2024-02-10T05:57:30.362 DEBUG:teuthology.orchestra.run.smithi028:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-02-10T05:57:30.362 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-02-10T05:57:31.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:30 smithi028 ceph-mon[66336]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:57:31.308 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:30 smithi028 ceph-mon[66336]: pgmap v424: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:31.321 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:30 smithi192 ceph-mon[52881]: from='mgr.34104 172.21.15.28:0/556492157' entity='mgr.smithi028.ucuqfx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:57:31.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:30 smithi192 ceph-mon[52881]: pgmap v424: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T05:57:31.999 INFO:teuthology.orchestra.run.smithi028.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-02-10T05:57:32.257 INFO:teuthology.orchestra.run.smithi028.stderr:+ [[ 6 -gt 2 ]] 2024-02-10T05:57:32.882 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T05:57:32.892 INFO:tasks.cephadm:Teardown begin 2024-02-10T05:57:32.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:57:32.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:57:32.960 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T05:57:32.961 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T05:57:32.982 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T05:57:33.018 INFO:tasks.cephadm:Stopping all daemons... 2024-02-10T05:57:33.018 INFO:tasks.cephadm.mon.smithi028:Stopping mon.smithi028... 2024-02-10T05:57:33.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi028 2024-02-10T05:57:33.158 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:33 smithi028 systemd[1]: Stopping Ceph mon.smithi028 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:57:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:33 smithi192 ceph-mon[52881]: pgmap v425: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T05:57:33.322 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:33 smithi192 ceph-mon[52881]: from='client.? 172.21.15.28:0/2140806727' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-02-10T05:57:33.467 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:33 smithi028 ceph-mon[66336]: pgmap v425: 97 pgs: 97 active+clean; 217 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T05:57:33.467 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:33 smithi028 ceph-mon[66336]: from='client.? 172.21.15.28:0/2140806727' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-02-10T05:57:33.467 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:33 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028[66332]: 2024-02-10T05:57:33.340+0000 7f3991fcb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi028 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:57:33.467 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:33 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028[66332]: 2024-02-10T05:57:33.340+0000 7f3991fcb700 -1 mon.smithi028@0(leader) e3 *** Got Signal Terminated *** 2024-02-10T05:57:33.808 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:33 smithi028 podman[111805]: 2024-02-10 05:57:33.606029028 +0000 UTC m=+0.387768543 container died 57f180a5eb092445eed580f154079b0915072f1725a7107427e3c401a16b5a4a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-10T05:57:34.208 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:33 smithi028 podman[111805]: 2024-02-10 05:57:33.982607991 +0000 UTC m=+0.764347508 container cleanup 57f180a5eb092445eed580f154079b0915072f1725a7107427e3c401a16b5a4a (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2) 2024-02-10T05:57:34.208 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Feb 10 05:57:33 smithi028 bash[111805]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi028 2024-02-10T05:57:34.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi028.service' 2024-02-10T05:57:34.397 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:57:34.397 INFO:tasks.cephadm.mon.smithi028:Stopped mon.smithi028 2024-02-10T05:57:34.397 INFO:tasks.cephadm.mon.smithi192:Stopping mon.smithi192... 2024-02-10T05:57:34.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi192 2024-02-10T05:57:34.708 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:34 smithi192 systemd[1]: Stopping Ceph mon.smithi192 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:57:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192[52877]: 2024-02-10T05:57:34.706+0000 7f007e922700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi192 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:57:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192[52877]: 2024-02-10T05:57:34.706+0000 7f007e922700 -1 mon.smithi192@1(peon) e3 *** Got Signal Terminated *** 2024-02-10T05:57:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:34 smithi192 podman[77802]: 2024-02-10 05:57:34.809673895 +0000 UTC m=+0.222331886 container stop 89a15b946f090f55885669e9d58a0efe49187c44f158eb4c7eb0ff5643740f50 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-10T05:57:35.072 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:34 smithi192 podman[77802]: 2024-02-10 05:57:34.835380276 +0000 UTC m=+0.248038309 container died 89a15b946f090f55885669e9d58a0efe49187c44f158eb4c7eb0ff5643740f50 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:57:35.467 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:35 smithi192 podman[77802]: 2024-02-10 05:57:35.290844688 +0000 UTC m=+0.703502676 container cleanup 89a15b946f090f55885669e9d58a0efe49187c44f158eb4c7eb0ff5643740f50 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:57:35.467 INFO:journalctl@ceph.mon.smithi192.smithi192.stdout:Feb 10 05:57:35 smithi192 bash[77802]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-mon-smithi192 2024-02-10T05:57:35.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@mon.smithi192.service' 2024-02-10T05:57:35.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:57:35.721 INFO:tasks.cephadm.mon.smithi192:Stopped mon.smithi192 2024-02-10T05:57:35.721 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-10T05:57:35.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.0 2024-02-10T05:57:36.019 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:35 smithi028 systemd[1]: Stopping Ceph osd.0 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:57:36.308 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:36 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0[72487]: 2024-02-10T05:57:36.018+0000 7f478ffd3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:57:36.309 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:36 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0[72487]: 2024-02-10T05:57:36.019+0000 7f478ffd3700 -1 osd.0 165 *** Got signal Terminated *** 2024-02-10T05:57:36.309 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:36 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0[72487]: 2024-02-10T05:57:36.019+0000 7f478ffd3700 -1 osd.0 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:57:41.809 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:41 smithi028 podman[111940]: 2024-02-10 05:57:41.529936281 +0000 UTC m=+5.612008306 container died 13cfdd85a4ed935dd9633f105cf572e42ff86707adec2dc8ddc6f2e629a3f0ef (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-10T05:57:42.163 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:41 smithi028 podman[111940]: 2024-02-10 05:57:41.962149112 +0000 UTC m=+6.044221134 container cleanup 13cfdd85a4ed935dd9633f105cf572e42ff86707adec2dc8ddc6f2e629a3f0ef (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0, org.label-schema.build-date=20240131, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-10T05:57:42.163 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:41 smithi028 bash[111940]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0 2024-02-10T05:57:42.558 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:42 smithi028 podman[111958]: 2024-02-10 05:57:42.163850524 +0000 UTC m=+0.638651945 container remove 13cfdd85a4ed935dd9633f105cf572e42ff86707adec2dc8ddc6f2e629a3f0ef (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-10T05:57:42.826 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:42 smithi028 podman[112052]: 2024-02-10 05:57:42.470942737 +0000 UTC m=+0.017724383 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:57:42.827 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:42 smithi028 podman[112052]: 2024-02-10 05:57:42.589851736 +0000 UTC m=+0.136633414 container create d7636a357255e249a1d78359feb17e44aceca3bec66c6d0a8b3f175b73d21f76 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-02-10T05:57:42.827 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:42 smithi028 podman[112052]: 2024-02-10 05:57:42.826530906 +0000 UTC m=+0.373312539 container init d7636a357255e249a1d78359feb17e44aceca3bec66c6d0a8b3f175b73d21f76 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-10T05:57:43.125 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:42 smithi028 podman[112052]: 2024-02-10 05:57:42.830256028 +0000 UTC m=+0.377037662 container start d7636a357255e249a1d78359feb17e44aceca3bec66c6d0a8b3f175b73d21f76 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS) 2024-02-10T05:57:43.125 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:42 smithi028 podman[112052]: 2024-02-10 05:57:42.917301669 +0000 UTC m=+0.464083305 container attach d7636a357255e249a1d78359feb17e44aceca3bec66c6d0a8b3f175b73d21f76 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:57:43.437 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:43 smithi028 podman[112052]: 2024-02-10 05:57:43.12581332 +0000 UTC m=+0.672594960 container died d7636a357255e249a1d78359feb17e44aceca3bec66c6d0a8b3f175b73d21f76 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=) 2024-02-10T05:57:43.689 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 10 05:57:43 smithi028 podman[112078]: 2024-02-10 05:57:43.687562161 +0000 UTC m=+0.551636632 container remove d7636a357255e249a1d78359feb17e44aceca3bec66c6d0a8b3f175b73d21f76 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-02-10T05:57:43.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.0.service' 2024-02-10T05:57:43.747 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:57:43.748 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-10T05:57:43.748 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-10T05:57:43.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.1 2024-02-10T05:57:44.058 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:43 smithi028 systemd[1]: Stopping Ceph osd.1 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:57:44.559 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:44 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:57:44.086+0000 7f47c8c4f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:57:44.559 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:44 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:57:44.086+0000 7f47c8c4f700 -1 osd.1 165 *** Got signal Terminated *** 2024-02-10T05:57:44.559 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:44 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1[75039]: 2024-02-10T05:57:44.086+0000 7f47c8c4f700 -1 osd.1 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:57:49.858 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:49 smithi028 podman[112158]: 2024-02-10 05:57:49.595452734 +0000 UTC m=+5.619134727 container died 84c53ce0260839fdca2552fb6c5eae8403c98f69f238374e7019823ad4db23f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-10T05:57:50.199 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:49 smithi028 podman[112158]: 2024-02-10 05:57:49.948991584 +0000 UTC m=+5.972673567 container cleanup 84c53ce0260839fdca2552fb6c5eae8403c98f69f238374e7019823ad4db23f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2) 2024-02-10T05:57:50.200 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:49 smithi028 bash[112158]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1 2024-02-10T05:57:50.500 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:50 smithi028 podman[112208]: 2024-02-10 05:57:50.200147814 +0000 UTC m=+0.603636277 container remove 84c53ce0260839fdca2552fb6c5eae8403c98f69f238374e7019823ad4db23f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:57:50.761 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:50 smithi028 podman[112298]: 2024-02-10 05:57:50.500985439 +0000 UTC m=+0.110815939 container create b10253fd957740cfd151574ff5de7707ed42fd9a7ed5917877a00b1a75ceefd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-10T05:57:50.761 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:50 smithi028 podman[112298]: 2024-02-10 05:57:50.408382184 +0000 UTC m=+0.018212692 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:57:50.762 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:50 smithi028 podman[112298]: 2024-02-10 05:57:50.686862363 +0000 UTC m=+0.296692873 container init b10253fd957740cfd151574ff5de7707ed42fd9a7ed5917877a00b1a75ceefd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:57:50.762 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:50 smithi028 podman[112298]: 2024-02-10 05:57:50.690916457 +0000 UTC m=+0.300746958 container start b10253fd957740cfd151574ff5de7707ed42fd9a7ed5917877a00b1a75ceefd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-10T05:57:50.762 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:50 smithi028 podman[112298]: 2024-02-10 05:57:50.761664103 +0000 UTC m=+0.371494622 container attach b10253fd957740cfd151574ff5de7707ed42fd9a7ed5917877a00b1a75ceefd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0) 2024-02-10T05:57:51.058 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:50 smithi028 podman[112298]: 2024-02-10 05:57:50.989392219 +0000 UTC m=+0.599222731 container died b10253fd957740cfd151574ff5de7707ed42fd9a7ed5917877a00b1a75ceefd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, ceph=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-10T05:57:51.534 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 10 05:57:51 smithi028 podman[112323]: 2024-02-10 05:57:51.532079538 +0000 UTC m=+0.531589445 container remove b10253fd957740cfd151574ff5de7707ed42fd9a7ed5917877a00b1a75ceefd6 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-290212a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:57:51.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.1.service' 2024-02-10T05:57:51.592 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:57:51.592 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-10T05:57:51.593 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-10T05:57:51.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.2 2024-02-10T05:57:51.808 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:51 smithi028 systemd[1]: Stopping Ceph osd.2 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:57:52.308 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:51 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2[77736]: 2024-02-10T05:57:51.942+0000 7f0575385700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:57:52.309 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:51 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2[77736]: 2024-02-10T05:57:51.942+0000 7f0575385700 -1 osd.2 165 *** Got signal Terminated *** 2024-02-10T05:57:52.309 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:51 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2[77736]: 2024-02-10T05:57:51.942+0000 7f0575385700 -1 osd.2 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:57:57.720 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:57 smithi028 podman[112402]: 2024-02-10 05:57:57.429114539 +0000 UTC m=+5.611558056 container died a45f0e24ae449906f47b8a167169e276435a0739e458c6ec2206d8b057332d03 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=main-290212a) 2024-02-10T05:57:58.043 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:57 smithi028 podman[112402]: 2024-02-10 05:57:57.818126935 +0000 UTC m=+6.000570450 container cleanup a45f0e24ae449906f47b8a167169e276435a0739e458c6ec2206d8b057332d03 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-10T05:57:58.043 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:57 smithi028 bash[112402]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2 2024-02-10T05:57:58.308 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:58 smithi028 podman[112419]: 2024-02-10 05:57:58.073318388 +0000 UTC m=+0.642520790 container remove a45f0e24ae449906f47b8a167169e276435a0739e458c6ec2206d8b057332d03 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2, ceph=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:57:58.642 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:58 smithi028 podman[112512]: 2024-02-10 05:57:58.294519418 +0000 UTC m=+0.018057504 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:57:58.642 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:58 smithi028 podman[112512]: 2024-02-10 05:57:58.395297376 +0000 UTC m=+0.118835462 container create 14b2f38b43747bd4864f545bd7673bab8a4abd576cf0e1595a9d37f026c4e38b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-10T05:57:58.642 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:58 smithi028 podman[112512]: 2024-02-10 05:57:58.639694565 +0000 UTC m=+0.363232655 container init 14b2f38b43747bd4864f545bd7673bab8a4abd576cf0e1595a9d37f026c4e38b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-10T05:57:58.642 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:58 smithi028 podman[112512]: 2024-02-10 05:57:58.642847312 +0000 UTC m=+0.366385403 container start 14b2f38b43747bd4864f545bd7673bab8a4abd576cf0e1595a9d37f026c4e38b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-10T05:57:58.939 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:58 smithi028 podman[112512]: 2024-02-10 05:57:58.731211154 +0000 UTC m=+0.454749254 container attach 14b2f38b43747bd4864f545bd7673bab8a4abd576cf0e1595a9d37f026c4e38b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240131, ceph=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD) 2024-02-10T05:57:59.234 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:58 smithi028 podman[112512]: 2024-02-10 05:57:58.940078429 +0000 UTC m=+0.663616528 container died 14b2f38b43747bd4864f545bd7673bab8a4abd576cf0e1595a9d37f026c4e38b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-290212a, org.label-schema.schema-version=1.0) 2024-02-10T05:57:59.488 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 10 05:57:59 smithi028 podman[112538]: 2024-02-10 05:57:59.478158302 +0000 UTC m=+0.527446888 container remove 14b2f38b43747bd4864f545bd7673bab8a4abd576cf0e1595a9d37f026c4e38b (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-2-deactivate, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0) 2024-02-10T05:57:59.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.2.service' 2024-02-10T05:57:59.533 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:57:59.533 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-10T05:57:59.534 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-10T05:57:59.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.3 2024-02-10T05:57:59.808 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:57:59 smithi028 systemd[1]: Stopping Ceph osd.3 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:58:00.309 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:57:59 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3[80329]: 2024-02-10T05:57:59.880+0000 7f8c5de6a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:58:00.309 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:57:59 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3[80329]: 2024-02-10T05:57:59.880+0000 7f8c5de6a700 -1 osd.3 165 *** Got signal Terminated *** 2024-02-10T05:58:00.309 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:57:59 smithi028 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3[80329]: 2024-02-10T05:57:59.880+0000 7f8c5de6a700 -1 osd.3 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:58:02.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:02 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:02.247+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:02.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:02 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:02.309+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:03.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:03 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:03.236+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:03.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:03 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:03.286+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:04.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:04 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:04.250+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:04.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:04 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:04.235+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:04.572 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:04 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:04.356+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:05.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:04 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:04.721+0000 7f1ce4851700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:40.604210+0000 front 2024-02-10T05:57:40.604358+0000 (oldest deadline 2024-02-10T05:58:04.104201+0000) 2024-02-10T05:58:05.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:05 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:05.263+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:05.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:05 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:05.242+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:05.572 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:05 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:05.403+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:05.657 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:58:05 smithi028 podman[112617]: 2024-02-10 05:58:05.403341553 +0000 UTC m=+5.647966795 container died 25f4cf94969eca9d6ac3da9d7362c35a8fe2151dfec3a497c81199f0994ef3f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-10T05:58:05.924 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:58:05 smithi028 podman[112617]: 2024-02-10 05:58:05.731142884 +0000 UTC m=+5.975768172 container cleanup 25f4cf94969eca9d6ac3da9d7362c35a8fe2151dfec3a497c81199f0994ef3f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-10T05:58:05.924 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:58:05 smithi028 bash[112617]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3 2024-02-10T05:58:06.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:05 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:05.693+0000 7f1ce4851700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:40.604210+0000 front 2024-02-10T05:57:40.604358+0000 (oldest deadline 2024-02-10T05:58:04.104201+0000) 2024-02-10T05:58:06.308 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:58:05 smithi028 podman[112668]: 2024-02-10 05:58:05.923869214 +0000 UTC m=+0.520725009 container remove 25f4cf94969eca9d6ac3da9d7362c35a8fe2151dfec3a497c81199f0994ef3f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-10T05:58:06.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:06 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:06.229+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:06.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:06 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:06.288+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:06.572 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:06 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:06.358+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:06.635 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:58:06 smithi028 podman[112755]: 2024-02-10 05:58:06.24519033 +0000 UTC m=+0.017817326 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:58:06.635 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:58:06 smithi028 podman[112755]: 2024-02-10 05:58:06.349760288 +0000 UTC m=+0.122387274 container create 88400acf58c2ace5e7536ed87cc2e311ddddb807c720894a24236d0ff1692ce8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240131) 2024-02-10T05:58:06.635 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:58:06 smithi028 podman[112755]: 2024-02-10 05:58:06.560293926 +0000 UTC m=+0.332920912 container init 88400acf58c2ace5e7536ed87cc2e311ddddb807c720894a24236d0ff1692ce8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:58:06.635 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:58:06 smithi028 podman[112755]: 2024-02-10 05:58:06.56379534 +0000 UTC m=+0.336422327 container start 88400acf58c2ace5e7536ed87cc2e311ddddb807c720894a24236d0ff1692ce8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-02-10T05:58:06.635 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:58:06 smithi028 podman[112755]: 2024-02-10 05:58:06.635274714 +0000 UTC m=+0.407901706 container attach 88400acf58c2ace5e7536ed87cc2e311ddddb807c720894a24236d0ff1692ce8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-290212a) 2024-02-10T05:58:07.058 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 10 05:58:06 smithi028 podman[112755]: 2024-02-10 05:58:06.851697664 +0000 UTC m=+0.624324643 container died 88400acf58c2ace5e7536ed87cc2e311ddddb807c720894a24236d0ff1692ce8 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, ceph=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-10T05:58:07.071 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:06 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:06.699+0000 7f1ce4851700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:40.604210+0000 front 2024-02-10T05:57:40.604358+0000 (oldest deadline 2024-02-10T05:58:04.104201+0000) 2024-02-10T05:58:07.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.3.service' 2024-02-10T05:58:07.487 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:58:07.487 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-10T05:58:07.487 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-10T05:58:07.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.4 2024-02-10T05:58:07.517 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:07 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:07.331+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:07.517 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:07 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:07.243+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:07.518 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:07 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:07.279+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:07.801 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:07 smithi192 systemd[1]: Stopping Ceph osd.4 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:58:07.801 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:07 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:07.699+0000 7f1ce4851700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:40.604210+0000 front 2024-02-10T05:57:40.604358+0000 (oldest deadline 2024-02-10T05:58:04.104201+0000) 2024-02-10T05:58:07.801 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:07 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:07.800+0000 7f1cec265700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:58:08.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:07 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:07.800+0000 7f1cec265700 -1 osd.4 165 *** Got signal Terminated *** 2024-02-10T05:58:08.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:07 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:07.800+0000 7f1cec265700 -1 osd.4 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:58:08.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:08 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:08.253+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:08.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:08 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:08.278+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:08.572 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:08 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:08.292+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:09.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:08 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:08.747+0000 7f1ce4851700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:40.604210+0000 front 2024-02-10T05:57:40.604358+0000 (oldest deadline 2024-02-10T05:58:04.104201+0000) 2024-02-10T05:58:09.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:09 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:09.217+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:09.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:09 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:09.302+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:09.572 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:09 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:09.300+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:10.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:09 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:09.705+0000 7f1ce4851700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:40.604210+0000 front 2024-02-10T05:57:40.604358+0000 (oldest deadline 2024-02-10T05:58:04.104201+0000) 2024-02-10T05:58:10.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:10 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:10.198+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:10.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:10 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:10.346+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:10.572 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:10 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:10.273+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:11.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:10 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:10.716+0000 7f1ce4851700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:40.604210+0000 front 2024-02-10T05:57:40.604358+0000 (oldest deadline 2024-02-10T05:58:04.104201+0000) 2024-02-10T05:58:11.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:10 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:10.716+0000 7f1ce4851700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.805556+0000 front 2024-02-10T05:57:48.805583+0000 (oldest deadline 2024-02-10T05:58:09.905316+0000) 2024-02-10T05:58:11.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:11 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:11.165+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:11.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:11 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:11.387+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:11.572 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:11 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:11.264+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:12.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:11 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:11.698+0000 7f1ce4851700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:40.604210+0000 front 2024-02-10T05:57:40.604358+0000 (oldest deadline 2024-02-10T05:58:04.104201+0000) 2024-02-10T05:58:12.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:11 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:11.698+0000 7f1ce4851700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.805556+0000 front 2024-02-10T05:57:48.805583+0000 (oldest deadline 2024-02-10T05:58:09.905316+0000) 2024-02-10T05:58:12.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:12 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:12.203+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:12.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:12 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:12.203+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:12.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:12 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:12.383+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:12.572 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:12 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:12.298+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:13.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:12 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:12.689+0000 7f1ce4851700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:40.604210+0000 front 2024-02-10T05:57:40.604358+0000 (oldest deadline 2024-02-10T05:58:04.104201+0000) 2024-02-10T05:58:13.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:12 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4[57836]: 2024-02-10T05:58:12.689+0000 7f1ce4851700 -1 osd.4 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.805556+0000 front 2024-02-10T05:57:48.805583+0000 (oldest deadline 2024-02-10T05:58:09.905316+0000) 2024-02-10T05:58:13.572 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:13 smithi192 podman[77999]: 2024-02-10 05:58:13.315681149 +0000 UTC m=+5.634412485 container died b15b2733ec773aed85f8a85e156e9d722f991cbafb81a9740ca2e0293a9e4aad (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:58:13.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:13 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:13.362+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:13.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:13 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:13.159+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:13.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:13 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:13.159+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:13.573 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:13 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:13.326+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:13.573 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:13 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:13.326+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:13.970 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:13 smithi192 podman[77999]: 2024-02-10 05:58:13.760181514 +0000 UTC m=+6.078912853 container cleanup b15b2733ec773aed85f8a85e156e9d722f991cbafb81a9740ca2e0293a9e4aad (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-10T05:58:13.970 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:13 smithi192 bash[77999]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4 2024-02-10T05:58:14.322 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:13 smithi192 podman[78018]: 2024-02-10 05:58:13.970290609 +0000 UTC m=+0.650115842 container remove b15b2733ec773aed85f8a85e156e9d722f991cbafb81a9740ca2e0293a9e4aad (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-10T05:58:14.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:14 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:14.140+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:14.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:14 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:14.140+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:14.601 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:14 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:14.389+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:14.602 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:14 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:14.389+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.467720+0000 front 2024-02-10T05:57:47.467676+0000 (oldest deadline 2024-02-10T05:58:13.367709+0000) 2024-02-10T05:58:14.602 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:14 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:14.350+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:14.602 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:14 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:14.350+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:14.602 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:14 smithi192 podman[78113]: 2024-02-10 05:58:14.272060124 +0000 UTC m=+0.018018935 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:58:14.602 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:14 smithi192 podman[78113]: 2024-02-10 05:58:14.379537425 +0000 UTC m=+0.125496216 container create bc94dc040e02d0f10ac15e631001f6b3b3cf67fa3ef9e72ae9f98c7cf0e76587 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-10T05:58:14.602 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:14 smithi192 podman[78113]: 2024-02-10 05:58:14.598781872 +0000 UTC m=+0.344740677 container init bc94dc040e02d0f10ac15e631001f6b3b3cf67fa3ef9e72ae9f98c7cf0e76587 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-10T05:58:14.890 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:14 smithi192 podman[78113]: 2024-02-10 05:58:14.601940968 +0000 UTC m=+0.347899769 container start bc94dc040e02d0f10ac15e631001f6b3b3cf67fa3ef9e72ae9f98c7cf0e76587 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a) 2024-02-10T05:58:14.890 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:14 smithi192 podman[78113]: 2024-02-10 05:58:14.699019086 +0000 UTC m=+0.444977896 container attach bc94dc040e02d0f10ac15e631001f6b3b3cf67fa3ef9e72ae9f98c7cf0e76587 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-10T05:58:15.193 INFO:journalctl@ceph.osd.4.smithi192.stdout:Feb 10 05:58:14 smithi192 podman[78113]: 2024-02-10 05:58:14.89107292 +0000 UTC m=+0.637031731 container died bc94dc040e02d0f10ac15e631001f6b3b3cf67fa3ef9e72ae9f98c7cf0e76587 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-10T05:58:15.194 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:15 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:15.118+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:15.194 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:15 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:15.118+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:15.477 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:15 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:15.408+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:15.477 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:15 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:15.408+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.467720+0000 front 2024-02-10T05:57:47.467676+0000 (oldest deadline 2024-02-10T05:58:13.367709+0000) 2024-02-10T05:58:15.478 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:15 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:15.336+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:15.478 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:15 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:15.336+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:15.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.4.service' 2024-02-10T05:58:15.578 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:58:15.578 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-10T05:58:15.578 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-10T05:58:15.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.5 2024-02-10T05:58:15.734 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:15 smithi192 systemd[1]: Stopping Ceph osd.5 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:58:16.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:15 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:15.922+0000 7f31bb64b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:58:16.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:15 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:15.922+0000 7f31bb64b700 -1 osd.5 165 *** Got signal Terminated *** 2024-02-10T05:58:16.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:15 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:15.922+0000 7f31bb64b700 -1 osd.5 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:58:16.424 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:16 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:16.304+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:16.424 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:16 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:16.304+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:16.424 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:16 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:16.120+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:16.424 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:16 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:16.120+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:16.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:16 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:16.423+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:16.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:16 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:16.423+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.467720+0000 front 2024-02-10T05:57:47.467676+0000 (oldest deadline 2024-02-10T05:58:13.367709+0000) 2024-02-10T05:58:17.458 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:17 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:17.334+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:17.458 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:17 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:17.334+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:17.458 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:17 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:17.116+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:17.458 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:17 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:17.116+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:17.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:17 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:17.458+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:17.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:17 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:17.458+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.467720+0000 front 2024-02-10T05:57:47.467676+0000 (oldest deadline 2024-02-10T05:58:13.367709+0000) 2024-02-10T05:58:18.449 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:18.319+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:18.449 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:18.319+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:18.449 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:18.107+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:18.449 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:18.107+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:18.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:18.449+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:18.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:18.449+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.467720+0000 front 2024-02-10T05:57:47.467676+0000 (oldest deadline 2024-02-10T05:58:13.367709+0000) 2024-02-10T05:58:18.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:18 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:18.449+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:54.468806+0000 front 2024-02-10T05:57:54.468720+0000 (oldest deadline 2024-02-10T05:58:17.968887+0000) 2024-02-10T05:58:19.457 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:19 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:19.359+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:19.457 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:19 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:19.359+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:19.457 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:19 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:19.359+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:53.820480+0000 front 2024-02-10T05:57:53.820591+0000 (oldest deadline 2024-02-10T05:58:18.520445+0000) 2024-02-10T05:58:19.458 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:19 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:19.131+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:19.458 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:19 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:19.131+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:19.458 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:19 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:19.131+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:19.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:19 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:19.457+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:19.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:19 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:19.457+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.467720+0000 front 2024-02-10T05:57:47.467676+0000 (oldest deadline 2024-02-10T05:58:13.367709+0000) 2024-02-10T05:58:19.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:19 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:19.457+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:54.468806+0000 front 2024-02-10T05:57:54.468720+0000 (oldest deadline 2024-02-10T05:58:17.968887+0000) 2024-02-10T05:58:20.370 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:20 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:20.113+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:20.370 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:20 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:20.113+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:20.370 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:20 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:20.113+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:20.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:20 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:20.476+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:37.565683+0000 front 2024-02-10T05:57:37.565839+0000 (oldest deadline 2024-02-10T05:58:01.665834+0000) 2024-02-10T05:58:20.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:20 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:20.476+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.467720+0000 front 2024-02-10T05:57:47.467676+0000 (oldest deadline 2024-02-10T05:58:13.367709+0000) 2024-02-10T05:58:20.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:20 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5[60095]: 2024-02-10T05:58:20.476+0000 7f31b3c37700 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:54.468806+0000 front 2024-02-10T05:57:54.468720+0000 (oldest deadline 2024-02-10T05:58:17.968887+0000) 2024-02-10T05:58:20.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:20 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:20.368+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:20.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:20 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:20.368+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:20.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:20 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:20.368+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:53.820480+0000 front 2024-02-10T05:57:53.820591+0000 (oldest deadline 2024-02-10T05:58:18.520445+0000) 2024-02-10T05:58:21.368 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:21 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:21.089+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:21.368 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:21 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:21.089+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:21.368 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:21 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:21.089+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:21.732 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:21 smithi192 podman[78220]: 2024-02-10 05:58:21.423369949 +0000 UTC m=+5.615538036 container died 000913de61d6317cefe868ee1de23cbeaaf84f4ad2f7b5358e36f3ad51ff18ba (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:58:21.732 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:21 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:21.367+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:21.732 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:21 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:21.367+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:21.732 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:21 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:21.367+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:53.820480+0000 front 2024-02-10T05:57:53.820591+0000 (oldest deadline 2024-02-10T05:58:18.520445+0000) 2024-02-10T05:58:22.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:21 smithi192 podman[78220]: 2024-02-10 05:58:21.923630111 +0000 UTC m=+6.115798201 container cleanup 000913de61d6317cefe868ee1de23cbeaaf84f4ad2f7b5358e36f3ad51ff18ba (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True, io.buildah.version=1.33.2) 2024-02-10T05:58:22.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:21 smithi192 bash[78220]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5 2024-02-10T05:58:22.387 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:22 smithi192 podman[78248]: 2024-02-10 05:58:22.157866149 +0000 UTC m=+0.734392110 container remove 000913de61d6317cefe868ee1de23cbeaaf84f4ad2f7b5358e36f3ad51ff18ba (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.2) 2024-02-10T05:58:22.387 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:22 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:22.115+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:22.387 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:22 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:22.115+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:22.387 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:22 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:22.115+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:22.711 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:22 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:22.386+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:22.711 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:22 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:22.386+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:22.711 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:22 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:22.386+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:53.820480+0000 front 2024-02-10T05:57:53.820591+0000 (oldest deadline 2024-02-10T05:58:18.520445+0000) 2024-02-10T05:58:22.711 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:22 smithi192 podman[78342]: 2024-02-10 05:58:22.397278762 +0000 UTC m=+0.017876507 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:58:22.711 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:22 smithi192 podman[78342]: 2024-02-10 05:58:22.508918056 +0000 UTC m=+0.129515781 container create 19efb3e878e124dfeeaf83adf24a580ad8b0417e4bdb13e87383c2366e5442bd (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-10T05:58:22.711 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:22 smithi192 podman[78342]: 2024-02-10 05:58:22.711245205 +0000 UTC m=+0.331842940 container init 19efb3e878e124dfeeaf83adf24a580ad8b0417e4bdb13e87383c2366e5442bd (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-10T05:58:23.004 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:22 smithi192 podman[78342]: 2024-02-10 05:58:22.714460127 +0000 UTC m=+0.335057856 container start 19efb3e878e124dfeeaf83adf24a580ad8b0417e4bdb13e87383c2366e5442bd (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-deactivate, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-10T05:58:23.004 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:22 smithi192 podman[78342]: 2024-02-10 05:58:22.786200407 +0000 UTC m=+0.406798151 container attach 19efb3e878e124dfeeaf83adf24a580ad8b0417e4bdb13e87383c2366e5442bd (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, ceph=True) 2024-02-10T05:58:23.297 INFO:journalctl@ceph.osd.5.smithi192.stdout:Feb 10 05:58:23 smithi192 podman[78342]: 2024-02-10 05:58:23.005499474 +0000 UTC m=+0.626097202 container died 19efb3e878e124dfeeaf83adf24a580ad8b0417e4bdb13e87383c2366e5442bd (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-290212a) 2024-02-10T05:58:23.298 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:23 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:23.101+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:23.298 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:23 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:23.101+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:23.298 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:23 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:23.101+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:23.556 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:23 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:23.409+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:23.556 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:23 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:23.409+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:23.556 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:23 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:23.409+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:53.820480+0000 front 2024-02-10T05:57:53.820591+0000 (oldest deadline 2024-02-10T05:58:18.520445+0000) 2024-02-10T05:58:23.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.5.service' 2024-02-10T05:58:23.621 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:58:23.621 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-10T05:58:23.621 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-10T05:58:23.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.6 2024-02-10T05:58:23.821 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:23 smithi192 systemd[1]: Stopping Ceph osd.6 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:58:24.322 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:23 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:23.959+0000 7f4bc53f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:58:24.322 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:23 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:23.959+0000 7f4bc53f2700 -1 osd.6 165 *** Got signal Terminated *** 2024-02-10T05:58:24.322 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:23 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:23.959+0000 7f4bc53f2700 -1 osd.6 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:58:24.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:24 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:24.119+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:24.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:24 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:24.119+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:24.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:24 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:24.119+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:24.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:24 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:24.376+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:24.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:24 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:24.376+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:24.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:24 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:24.376+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:53.820480+0000 front 2024-02-10T05:57:53.820591+0000 (oldest deadline 2024-02-10T05:58:18.520445+0000) 2024-02-10T05:58:25.401 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:25 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:25.104+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:25.402 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:25 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:25.104+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:25.402 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:25 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:25.104+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:25.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:25 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:25.402+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:25.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:25 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:25.402+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:25.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:25 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:25.402+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:53.820480+0000 front 2024-02-10T05:57:53.820591+0000 (oldest deadline 2024-02-10T05:58:18.520445+0000) 2024-02-10T05:58:26.374 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:26 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:26.079+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:26.375 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:26 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:26.079+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:26.375 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:26 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:26.079+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:26.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:26 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:26.374+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:26.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:26 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:26.374+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:26.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:26 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:26.374+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:53.820480+0000 front 2024-02-10T05:57:53.820591+0000 (oldest deadline 2024-02-10T05:58:18.520445+0000) 2024-02-10T05:58:26.722 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:26 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:26.374+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:00.221405+0000 front 2024-02-10T05:58:00.221364+0000 (oldest deadline 2024-02-10T05:58:26.121474+0000) 2024-02-10T05:58:27.395 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:27 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:27.104+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:27.396 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:27 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:27.104+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:27.396 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:27 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:27.104+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:27.396 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:27 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:27.104+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:01.160365+0000 front 2024-02-10T05:58:01.160517+0000 (oldest deadline 2024-02-10T05:58:26.460398+0000) 2024-02-10T05:58:27.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:27 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:27.395+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:27.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:27 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:27.395+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:27.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:27 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:27.395+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:53.820480+0000 front 2024-02-10T05:57:53.820591+0000 (oldest deadline 2024-02-10T05:58:18.520445+0000) 2024-02-10T05:58:27.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:27 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:27.395+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:00.221405+0000 front 2024-02-10T05:58:00.221364+0000 (oldest deadline 2024-02-10T05:58:26.121474+0000) 2024-02-10T05:58:28.382 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:28 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:28.108+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:28.382 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:28 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:28.108+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:28.382 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:28 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:28.108+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:28.382 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:28 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:28.108+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:01.160365+0000 front 2024-02-10T05:58:01.160517+0000 (oldest deadline 2024-02-10T05:58:26.460398+0000) 2024-02-10T05:58:28.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:28 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:28.381+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:39.516817+0000 front 2024-02-10T05:57:39.516831+0000 (oldest deadline 2024-02-10T05:58:04.216678+0000) 2024-02-10T05:58:28.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:28 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:28.381+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:48.718708+0000 front 2024-02-10T05:57:48.718825+0000 (oldest deadline 2024-02-10T05:58:12.719231+0000) 2024-02-10T05:58:28.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:28 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:28.381+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:53.820480+0000 front 2024-02-10T05:57:53.820591+0000 (oldest deadline 2024-02-10T05:58:18.520445+0000) 2024-02-10T05:58:28.822 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:28 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6[62371]: 2024-02-10T05:58:28.381+0000 7f4bbd9de700 -1 osd.6 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:00.221405+0000 front 2024-02-10T05:58:00.221364+0000 (oldest deadline 2024-02-10T05:58:26.121474+0000) 2024-02-10T05:58:29.528 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:29 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:29.125+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:29.528 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:29 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:29.125+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:29.528 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:29 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:29.125+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:29.528 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:29 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:29.125+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:01.160365+0000 front 2024-02-10T05:58:01.160517+0000 (oldest deadline 2024-02-10T05:58:26.460398+0000) 2024-02-10T05:58:29.793 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:29 smithi192 podman[78447]: 2024-02-10 05:58:29.528123866 +0000 UTC m=+5.688244746 container died c9776c8b4f21fd88b8f9889f42108a2624ef3ca2fbd0b7016d94c750a2c4e2e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD) 2024-02-10T05:58:30.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:29 smithi192 podman[78447]: 2024-02-10 05:58:29.892494625 +0000 UTC m=+6.052615495 container cleanup c9776c8b4f21fd88b8f9889f42108a2624ef3ca2fbd0b7016d94c750a2c4e2e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-10T05:58:30.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:29 smithi192 bash[78447]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6 2024-02-10T05:58:30.484 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:30 smithi192 podman[78496]: 2024-02-10 05:58:30.185634324 +0000 UTC m=+0.657589504 container remove c9776c8b4f21fd88b8f9889f42108a2624ef3ca2fbd0b7016d94c750a2c4e2e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-10T05:58:30.485 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:30 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:30.160+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:30.485 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:30 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:30.160+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:30.485 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:30 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:30.160+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:30.485 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:30 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:30.160+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:01.160365+0000 front 2024-02-10T05:58:01.160517+0000 (oldest deadline 2024-02-10T05:58:26.460398+0000) 2024-02-10T05:58:30.747 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:30 smithi192 podman[78589]: 2024-02-10 05:58:30.388292299 +0000 UTC m=+0.018090385 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:58:30.747 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:30 smithi192 podman[78589]: 2024-02-10 05:58:30.486350089 +0000 UTC m=+0.116148166 container create 6df88da6496a47a97fce9975b57033f6a9e805308f9d8f8b0136450103b1f2f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-10T05:58:30.747 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:30 smithi192 podman[78589]: 2024-02-10 05:58:30.747714837 +0000 UTC m=+0.377512915 container init 6df88da6496a47a97fce9975b57033f6a9e805308f9d8f8b0136450103b1f2f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.build-date=20240131, ceph=True) 2024-02-10T05:58:31.040 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:30 smithi192 podman[78589]: 2024-02-10 05:58:30.75115172 +0000 UTC m=+0.380949806 container start 6df88da6496a47a97fce9975b57033f6a9e805308f9d8f8b0136450103b1f2f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-deactivate, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-10T05:58:31.041 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:30 smithi192 podman[78589]: 2024-02-10 05:58:30.839247313 +0000 UTC m=+0.469045391 container attach 6df88da6496a47a97fce9975b57033f6a9e805308f9d8f8b0136450103b1f2f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-02-10T05:58:31.322 INFO:journalctl@ceph.osd.6.smithi192.stdout:Feb 10 05:58:31 smithi192 podman[78589]: 2024-02-10 05:58:31.041741679 +0000 UTC m=+0.671539756 container died 6df88da6496a47a97fce9975b57033f6a9e805308f9d8f8b0136450103b1f2f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-10T05:58:31.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:31 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:31.184+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:31.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:31 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:31.184+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:31.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:31 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:31.184+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:31.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:31 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:31.184+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:01.160365+0000 front 2024-02-10T05:58:01.160517+0000 (oldest deadline 2024-02-10T05:58:26.460398+0000) 2024-02-10T05:58:31.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.6.service' 2024-02-10T05:58:31.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:58:31.655 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-10T05:58:31.655 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-10T05:58:31.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.7 2024-02-10T05:58:31.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:31 smithi192 systemd[1]: Stopping Ceph osd.7 for cf1ef3cc-c7d5-11ee-95b8-87774f69a715... 2024-02-10T05:58:31.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:31 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:31.986+0000 7f16d1360700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:58:31.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:31 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:31.986+0000 7f16d1360700 -1 osd.7 165 *** Got signal Terminated *** 2024-02-10T05:58:31.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:31 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:31.986+0000 7f16d1360700 -1 osd.7 165 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:58:32.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:32 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:32.157+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:32.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:32 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:32.157+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:32.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:32 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:32.157+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:32.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:32 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:32.157+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:01.160365+0000 front 2024-02-10T05:58:01.160517+0000 (oldest deadline 2024-02-10T05:58:26.460398+0000) 2024-02-10T05:58:33.570 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:33 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:33.144+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:33.571 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:33 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:33.144+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:33.571 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:33 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:33.144+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:33.571 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:33 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:33.144+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:01.160365+0000 front 2024-02-10T05:58:01.160517+0000 (oldest deadline 2024-02-10T05:58:26.460398+0000) 2024-02-10T05:58:34.461 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:34.144+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:34.461 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:34.144+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:34.462 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:34.144+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:34.462 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:34.144+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:01.160365+0000 front 2024-02-10T05:58:01.160517+0000 (oldest deadline 2024-02-10T05:58:26.460398+0000) 2024-02-10T05:58:34.462 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:34 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:34.144+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.192:6804 osd.4 since back 2024-02-10T05:58:12.261732+0000 front 2024-02-10T05:58:12.261812+0000 (oldest deadline 2024-02-10T05:58:33.361701+0000) 2024-02-10T05:58:35.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:35 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:35.121+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:35.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:35 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:35.121+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:35.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:35 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:35.121+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:35.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:35 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:35.121+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:01.160365+0000 front 2024-02-10T05:58:01.160517+0000 (oldest deadline 2024-02-10T05:58:26.460398+0000) 2024-02-10T05:58:35.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:35 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:35.121+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.192:6804 osd.4 since back 2024-02-10T05:58:12.261732+0000 front 2024-02-10T05:58:12.261812+0000 (oldest deadline 2024-02-10T05:58:33.361701+0000) 2024-02-10T05:58:36.571 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:36 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:36.134+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-02-10T05:57:36.558026+0000 front 2024-02-10T05:57:36.558120+0000 (oldest deadline 2024-02-10T05:58:01.857928+0000) 2024-02-10T05:58:36.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:36 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:36.134+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6814 osd.1 since back 2024-02-10T05:57:47.758877+0000 front 2024-02-10T05:57:47.758835+0000 (oldest deadline 2024-02-10T05:58:11.858730+0000) 2024-02-10T05:58:36.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:36 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:36.134+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6822 osd.2 since back 2024-02-10T05:57:55.359563+0000 front 2024-02-10T05:57:55.359533+0000 (oldest deadline 2024-02-10T05:58:18.259437+0000) 2024-02-10T05:58:36.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:36 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:36.134+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.28:6830 osd.3 since back 2024-02-10T05:58:01.160365+0000 front 2024-02-10T05:58:01.160517+0000 (oldest deadline 2024-02-10T05:58:26.460398+0000) 2024-02-10T05:58:36.572 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:36 smithi192 ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7[64650]: 2024-02-10T05:58:36.134+0000 7f16cd954700 -1 osd.7 165 heartbeat_check: no reply from 172.21.15.192:6804 osd.4 since back 2024-02-10T05:58:12.261732+0000 front 2024-02-10T05:58:12.261812+0000 (oldest deadline 2024-02-10T05:58:33.361701+0000) 2024-02-10T05:58:37.662 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:37 smithi192 podman[78697]: 2024-02-10 05:58:37.407735907 +0000 UTC m=+5.533783603 container died 86dbef76e22dfa12abbc808c06ea14f26ad230c0af97146fa4b5493867dd97e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-10T05:58:37.938 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:37 smithi192 podman[78697]: 2024-02-10 05:58:37.736859252 +0000 UTC m=+5.862906957 container cleanup 86dbef76e22dfa12abbc808c06ea14f26ad230c0af97146fa4b5493867dd97e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-10T05:58:37.938 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:37 smithi192 bash[78697]: ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7 2024-02-10T05:58:38.322 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:37 smithi192 podman[78728]: 2024-02-10 05:58:37.938058785 +0000 UTC m=+0.532169657 container remove 86dbef76e22dfa12abbc808c06ea14f26ad230c0af97146fa4b5493867dd97e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-10T05:58:38.666 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:38 smithi192 podman[78819]: 2024-02-10 05:58:38.253610948 +0000 UTC m=+0.018848092 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:58:38.666 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:38 smithi192 podman[78819]: 2024-02-10 05:58:38.372381398 +0000 UTC m=+0.137618537 container create d033330b9a6913ddff3ddfb270ddc61a496b37182be776f99916be3ef4595d5d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240131) 2024-02-10T05:58:38.666 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:38 smithi192 podman[78819]: 2024-02-10 05:58:38.583069067 +0000 UTC m=+0.348306207 container init d033330b9a6913ddff3ddfb270ddc61a496b37182be776f99916be3ef4595d5d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD) 2024-02-10T05:58:38.666 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:38 smithi192 podman[78819]: 2024-02-10 05:58:38.586050884 +0000 UTC m=+0.351288032 container start d033330b9a6913ddff3ddfb270ddc61a496b37182be776f99916be3ef4595d5d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, ceph=True, org.label-schema.vendor=CentOS) 2024-02-10T05:58:38.666 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:38 smithi192 podman[78819]: 2024-02-10 05:58:38.66627897 +0000 UTC m=+0.431516119 container attach d033330b9a6913ddff3ddfb270ddc61a496b37182be776f99916be3ef4595d5d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-deactivate, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:58:39.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Feb 10 05:58:38 smithi192 podman[78819]: 2024-02-10 05:58:38.868477573 +0000 UTC m=+0.633714715 container died d033330b9a6913ddff3ddfb270ddc61a496b37182be776f99916be3ef4595d5d (image=quay.ceph.io/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715-osd-7-deactivate, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-10T05:58:39.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf1ef3cc-c7d5-11ee-95b8-87774f69a715@osd.7.service' 2024-02-10T05:58:39.517 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:58:39.518 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-10T05:58:39.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T06:01:37.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T06:01:57.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:01:57.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:01:57.317 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T06:01:57.318 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078/remote/smithi028/crash 2024-02-10T06:01:57.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/crash -- . 2024-02-10T06:01:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T06:01:57.367 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-02-10T06:01:57.368 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078/remote/smithi192/crash 2024-02-10T06:01:57.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/crash -- . 2024-02-10T06:01:57.407 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T06:01:57.407 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2024-02-10T06:01:57.408 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T06:01:57.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T06:01:57.439 INFO:teuthology.orchestra.run.smithi028.stdout:2024-02-10T05:38:25.393513+0000 mon.smithi028 (mon.0) 503 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:01:57.440 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-10T06:01:57.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[SEC\]' /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T06:01:57.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]' /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T06:01:57.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[WRN\]' /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T06:01:57.653 INFO:teuthology.orchestra.run.smithi028.stdout:2024-02-10T05:38:25.393513+0000 mon.smithi028 (mon.0) 503 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:01:57.653 INFO:tasks.cephadm:Compressing logs... 2024-02-10T06:01:57.653 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T06:01:57.695 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T06:01:57.720 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2024-02-10T06:01:57.720 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-10T06:01:57.720 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mds.foofs.smithi028.xhtowe.log 2024-02-10T06:01:57.725 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T06:01:57.725 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-10T06:01:57.725 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mds.foofs.smithi028.xhtowe.log: /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.3.log 2024-02-10T06:01:57.725 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mgr.smithi028.ucuqfx.log 2024-02-10T06:01:57.725 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.1.log: 89.2% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mds.foofs.smithi028.xhtowe.log.gz 2024-02-10T06:01:57.730 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mon.smithi028.log 2024-02-10T06:01:57.730 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mgr.smithi028.ucuqfx.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-10T06:01:57.730 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mon.smithi028.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T06:01:57.730 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.log 2024-02-10T06:01:57.730 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.cephadm.log: 85.6% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T06:01:57.730 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-10T06:01:57.730 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.log: 91.5% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T06:01:57.730 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T06:01:57.730 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T06:01:57.733 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T06:01:57.733 INFO:teuthology.orchestra.run.smithi192.stderr:: No such file or directory 2024-02-10T06:01:57.733 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.6.log 2024-02-10T06:01:57.733 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mgr.smithi192.hppzcm.log 2024-02-10T06:01:57.733 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mon.smithi192.log 2024-02-10T06:01:57.733 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mgr.smithi192.hppzcm.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T06:01:57.734 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.4.log 2024-02-10T06:01:57.734 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mon.smithi192.log: /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.7.log 2024-02-10T06:01:57.734 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.5.log 2024-02-10T06:01:57.734 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T06:01:57.734 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.5.log: 91.7% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T06:01:57.734 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.log 2024-02-10T06:01:57.734 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.cephadm.log: 90.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T06:01:57.734 INFO:teuthology.orchestra.run.smithi192.stderr: 85.6% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T06:01:57.734 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mds.foofs.smithi192.hdcbzw.log 2024-02-10T06:01:57.734 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T06:01:57.734 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mds.foofs.smithi192.hdcbzw.log: 87.6% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mds.foofs.smithi192.hdcbzw.log.gz 2024-02-10T06:01:57.735 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T06:01:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.2.log: /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-volume.log: 90.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T06:01:57.747 INFO:teuthology.orchestra.run.smithi028.stderr: 91.4% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T06:01:57.751 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-volume.log: 91.4% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T06:01:57.785 INFO:teuthology.orchestra.run.smithi192.stderr: 89.1% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mgr.smithi192.hppzcm.log.gz 2024-02-10T06:01:58.000 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mgr.smithi028.ucuqfx.log.gz 2024-02-10T06:01:58.229 INFO:teuthology.orchestra.run.smithi192.stderr: 92.7% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mon.smithi192.log.gz 2024-02-10T06:01:59.204 INFO:teuthology.orchestra.run.smithi028.stderr: 92.0% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-mon.smithi028.log.gz 2024-02-10T06:02:11.137 INFO:teuthology.orchestra.run.smithi028.stderr: 94.5% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-10T06:02:11.984 INFO:teuthology.orchestra.run.smithi192.stderr: 93.9% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.4.log.gz 2024-02-10T06:02:14.320 INFO:teuthology.orchestra.run.smithi192.stderr: 94.2% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.6.log.gz 2024-02-10T06:02:16.033 INFO:teuthology.orchestra.run.smithi028.stderr: 94.1% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-10T06:02:17.098 INFO:teuthology.orchestra.run.smithi028.stderr: 94.2% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-10T06:02:17.652 INFO:teuthology.orchestra.run.smithi028.stderr: 94.0% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.3.log.gz 2024-02-10T06:02:17.654 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-10T06:02:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m19.946s 2024-02-10T06:02:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:user 1m10.189s 2024-02-10T06:02:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m2.543s 2024-02-10T06:02:17.860 INFO:teuthology.orchestra.run.smithi192.stderr: 94.1% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.5.log.gz 2024-02-10T06:02:23.345 INFO:teuthology.orchestra.run.smithi192.stderr: 94.1% -- replaced with /var/log/ceph/cf1ef3cc-c7d5-11ee-95b8-87774f69a715/ceph-osd.7.log.gz 2024-02-10T06:02:23.347 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-02-10T06:02:23.347 INFO:teuthology.orchestra.run.smithi192.stderr:real 0m25.636s 2024-02-10T06:02:23.347 INFO:teuthology.orchestra.run.smithi192.stderr:user 1m14.127s 2024-02-10T06:02:23.347 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m2.958s 2024-02-10T06:02:23.347 INFO:tasks.cephadm:Archiving logs... 2024-02-10T06:02:23.348 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078/remote/smithi028/log 2024-02-10T06:02:23.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T06:02:32.904 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078/remote/smithi192/log 2024-02-10T06:02:32.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T06:02:44.070 INFO:tasks.cephadm:Removing cluster... 2024-02-10T06:02:44.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 --force 2024-02-10T06:02:44.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf1ef3cc-c7d5-11ee-95b8-87774f69a715 --force 2024-02-10T06:02:44.828 INFO:tasks.cephadm:Removing cephadm ... 2024-02-10T06:02:44.829 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T06:02:44.844 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T06:02:44.859 INFO:tasks.cephadm:Teardown complete 2024-02-10T06:02:44.859 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T06:02:44.869 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T06:02:44.869 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:02:44.887 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:02:44.898 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:02:44.903 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:02:44.903 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-02-10T06:02:44.903 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 +728us[ +762us] +/- 81ms 2024-02-10T06:02:44.904 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -931us[ -896us] +/- 80ms 2024-02-10T06:02:44.904 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 +497us[ +497us] +/- 92ms 2024-02-10T06:02:44.904 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:02:44.912 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:02:44.916 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:02:44.916 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-02-10T06:02:44.917 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 +710us[ +710us] +/- 81ms 2024-02-10T06:02:44.917 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -949us[ -900us] +/- 80ms 2024-02-10T06:02:44.917 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +467us[ +467us] +/- 92ms 2024-02-10T06:02:44.917 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:02:44.918 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T06:02:44.928 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T06:02:44.929 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T06:02:44.939 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T06:02:45.028 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T06:02:45.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T06:02:45.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:02:45.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T06:02:45.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:02:45.169 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T06:02:45.179 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T06:02:45.188 INFO:teuthology.task.internal:Duration was 2060.832563 seconds 2024-02-10T06:02:45.189 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T06:02:45.198 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T06:02:45.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T06:02:45.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T06:02:45.241 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T06:02:45.252 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T06:02:45.609 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T06:02:45.609 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-02-10T06:02:45.610 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T06:02:45.660 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-02-10T06:02:45.660 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T06:02:45.682 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T06:02:45.683 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T06:02:45.702 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T06:02:45.752 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T06:02:45.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T06:02:45.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T06:02:59.964 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T06:02:59.991 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T06:02:59.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T06:03:00.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T06:03:00.052 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T06:03:00.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:03:00.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:03:00.088 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-02-10T06:03:00.121 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-02-10T06:03:00.146 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:03:00.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:03:00.163 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:03:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:03:00.202 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T06:03:00.213 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T06:03:00.214 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078/remote/smithi028 2024-02-10T06:03:00.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T06:03:00.278 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078/remote/smithi192 2024-02-10T06:03:00.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T06:03:00.320 INFO:teuthology.task.internal:Removing archive directory... 2024-02-10T06:03:00.321 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T06:03:00.323 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T06:03:00.370 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T06:03:00.381 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T06:03:00.382 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T06:03:00.392 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T06:03:00.392 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T06:03:00.394 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T06:03:00.406 INFO:teuthology.orchestra.run.smithi028.stdout: 265293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 06:03 /home/ubuntu/cephtest 2024-02-10T06:03:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: 265672 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 06:03 /home/ubuntu/cephtest 2024-02-10T06:03:00.426 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T06:03:00.437 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T06:03:00.457 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T06:03:00.483 DEBUG:teuthology.nuke:shortname: smithi028 2024-02-10T06:03:00.483 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T06:03:00.506 DEBUG:teuthology.nuke:shortname: smithi192 2024-02-10T06:03:00.506 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T06:03:00.510 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:20:40.379279', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGQ/7PSiIHjpmgrRBmPojrxiYa0EX+4kultoEOa+dZsiOf8UfXHSVMYjDpXTN/mO09RFJpV1Qcu2JcxyfLp5Kok='} 2024-02-10T06:03:00.533 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555078', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:20:40.381044', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLzI7ZOpv/aTGxmsFdFemq4T22bYmOkLj0RD4x7ud+DYc1lBem6EmfdN9q5jEKT1ZWg0cUAsfNHs6o2zwxjn4wE='} 2024-02-10T06:03:00.559 INFO:teuthology.orchestra.console.smithi028:Power off 2024-02-10T06:03:00.559 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T06:03:00.590 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-02-10T06:03:00.590 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:03:00.605 INFO:teuthology.orchestra.console.smithi192:Power off 2024-02-10T06:03:00.605 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T06:03:00.616 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-02-10T06:03:00.627 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2024-02-10T06:03:00.628 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:03:00.651 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-02-10T06:03:04.617 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:03:04.652 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:03:04.728 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-02-10T06:03:04.762 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-02-10T06:03:08.732 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:03:08.763 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:03:08.845 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-02-10T06:03:08.874 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2024-02-10T06:03:08.946 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-02-10T06:03:08.975 INFO:teuthology.orchestra.console.smithi192:Power off completed 2024-02-10T06:03:09.079 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2060.8325634002686 failure_reason: '"2024-02-10T05:38:25.393513+0000 mon.smithi028 (mon.0) 503 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' owner: scheduled_yuriw@teuthology success: false 2024-02-10T06:03:09.080 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T06:03:09.164 INFO:teuthology.run:FAIL