2024-02-27T06:01:09.518 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-27T06:01:09.519 DEBUG:teuthology.run:Teuthology command: teuthology --name adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi --archive /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463 --description orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --owner scheduled_adking@teuthology --verbose -- /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463/orig.config.yaml 2024-02-27T06:01:09.554 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-27T06:01:09.642 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463 branch: wip-adk-testing-2024-02-26-1600 description: orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: adking@redhat.com first_in_suite: false job_id: '7574463' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-02-26-1600 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 635ee6850c57c98cc6ba932cade45f9cc015aa32 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 635ee6850c57c98cc6ba932cade45f9cc015aa32 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-adk-testing-2024-02-26-1600 sha1: 635ee6850c57c98cc6ba932cade45f9cc015aa32 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 3790 sha1: 635ee6850c57c98cc6ba932cade45f9cc015aa32 sleep_before_teardown: 0 subset: 1/14 suite: orch:cephadm suite_branch: wip-adk-testing-2024-02-26-1600 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 635ee6850c57c98cc6ba932cade45f9cc015aa32 targets: smithi131.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyMMU2WmPTrG0nEyQ2NmvUMXtb5sj/oZhk6zkIj6rVET3wh7KaevTVGclb2I20iC0f0mf/os1Vdr5fuzKhXUrU= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1EuQQJmp6qcOuc+G8GQiW/tx9dd8YBP2iomrl9Czw/K/0oLGOOuefBVJ8PZ2KxfCk/0AouJ1k7OKERdbVOv+k= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-27_05:08:20 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-02-27T06:01:09.642 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa; will attempt to use it 2024-02-27T06:01:09.643 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa/tasks 2024-02-27T06:01:09.643 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-27T06:01:09.644 INFO:teuthology.task.internal:Checking packages... 2024-02-27T06:01:09.667 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '635ee6850c57c98cc6ba932cade45f9cc015aa32' 2024-02-27T06:01:09.667 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-27T06:01:09.667 INFO:teuthology.packaging:ref: None 2024-02-27T06:01:09.667 INFO:teuthology.packaging:tag: None 2024-02-27T06:01:09.667 INFO:teuthology.packaging:branch: wip-adk-testing-2024-02-26-1600 2024-02-27T06:01:09.667 INFO:teuthology.packaging:sha1: 635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T06:01:09.667 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-02-26-1600 2024-02-27T06:01:09.808 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1736.g635ee685 2024-02-27T06:01:09.809 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-27T06:01:09.816 INFO:teuthology.task.internal:no buildpackages task found 2024-02-27T06:01:09.816 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-27T06:01:09.865 INFO:teuthology.task.internal:Saving configuration 2024-02-27T06:01:09.878 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-27T06:01:09.890 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-27T06:01:09.915 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-27 05:55:55.417639', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyMMU2WmPTrG0nEyQ2NmvUMXtb5sj/oZhk6zkIj6rVET3wh7KaevTVGclb2I20iC0f0mf/os1Vdr5fuzKhXUrU='} 2024-02-27T06:01:09.937 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-27 05:55:55.416283', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1EuQQJmp6qcOuc+G8GQiW/tx9dd8YBP2iomrl9Czw/K/0oLGOOuefBVJ8PZ2KxfCk/0AouJ1k7OKERdbVOv+k='} 2024-02-27T06:01:09.937 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-27T06:01:09.942 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-02-27T06:01:09.943 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-02-27T06:01:09.943 INFO:teuthology.run_tasks:Running task console_log... 2024-02-27T06:01:10.025 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f944c762160>, signals=[15]) 2024-02-27T06:01:10.025 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-27T06:01:10.031 INFO:teuthology.task.internal:Opening connections... 2024-02-27T06:01:10.031 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:01:10.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T06:01:10.123 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-02-27T06:01:10.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T06:01:10.210 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-27T06:01:10.216 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2024-02-27T06:01:10.234 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2024-02-27T06:01:10.234 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-02-27T06:01:10.290 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="9" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="9" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el9" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:LOGO="fedora-logo-icon" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-27T06:01:10.291 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-27T06:01:10.292 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-02-27T06:01:10.311 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-02-27T06:01:10.328 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-02-27T06:01:10.328 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-02-27T06:01:10.386 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2024-02-27T06:01:10.386 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2024-02-27T06:01:10.386 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2024-02-27T06:01:10.386 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2024-02-27T06:01:10.386 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2024-02-27T06:01:10.386 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2024-02-27T06:01:10.386 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-27T06:01:10.386 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2024-02-27T06:01:10.386 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2024-02-27T06:01:10.386 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-27T06:01:10.386 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2024-02-27T06:01:10.387 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-27T06:01:10.387 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-27T06:01:10.387 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-27T06:01:10.387 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-02-27T06:01:10.410 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-27T06:01:10.419 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-27T06:01:10.490 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-27T06:01:10.490 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2024-02-27T06:01:10.493 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-02-27T06:01:10.509 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-27T06:01:10.514 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-27T06:01:10.514 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2024-02-27T06:01:10.549 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-02-27T06:01:10.702 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-27T06:01:10.757 INFO:teuthology.run_tasks:Running task kernel... 2024-02-27T06:01:10.772 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-27T06:01:10.772 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-27T06:01:10.773 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-27T06:01:10.773 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-27T06:01:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T06:01:10.789 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-02-27T06:01:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T06:01:10.790 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-02-27T06:01:10.846 INFO:teuthology.orchestra.run.smithi131.stdout:5.14.0-361.el9.x86_64 2024-02-27T06:01:10.847 INFO:teuthology.task.kernel:Running kernel on smithi131: 5.14.0-361.el9.x86_64 2024-02-27T06:01:10.847 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum install -y kernel 2024-02-27T06:01:10.848 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-361.el9.x86_64 2024-02-27T06:01:10.848 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.14.0-361.el9.x86_64 2024-02-27T06:01:10.848 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2024-02-27T06:01:12.953 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS-9 - AppStream 16 MB/s | 8.1 MB 00:00 2024-02-27T06:01:12.970 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 16 MB/s | 8.1 MB 00:00 2024-02-27T06:01:14.570 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 8.6 MB/s | 2.5 MB 00:00 2024-02-27T06:01:14.579 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS-9 - BaseOS 8.3 MB/s | 2.5 MB 00:00 2024-02-27T06:01:15.653 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-02-27T06:01:18.803 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-02-27T06:01:35.886 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-02-27T06:01:42.495 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.1 MB 00:02 2024-02-27T06:01:44.155 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-02-27T06:01:45.848 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2024-02-27T06:01:50.714 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-02-27T06:01:52.942 INFO:teuthology.orchestra.run.smithi131.stderr:Modular dependency problems: 2024-02-27T06:01:52.942 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-02-27T06:01:52.942 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 1: conflicting requests 2024-02-27T06:01:52.942 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240219154654:20125149.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.942 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 2: conflicting requests 2024-02-27T06:01:52.942 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.942 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 3: conflicting requests 2024-02-27T06:01:52.942 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 4: conflicting requests 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 5: conflicting requests 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 6: conflicting requests 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 7: conflicting requests 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 8: conflicting requests 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 9: conflicting requests 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 10: conflicting requests 2024-02-27T06:01:52.943 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.944 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 11: conflicting requests 2024-02-27T06:01:52.944 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.944 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 12: conflicting requests 2024-02-27T06:01:52.944 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.944 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 13: conflicting requests 2024-02-27T06:01:52.944 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.944 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 14: conflicting requests 2024-02-27T06:01:52.944 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 15: conflicting requests 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 16: conflicting requests 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 17: conflicting requests 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 18: conflicting requests 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 19: conflicting requests 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 20: conflicting requests 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 21: conflicting requests 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 22: conflicting requests 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.945 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 23: conflicting requests 2024-02-27T06:01:52.946 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.946 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 24: conflicting requests 2024-02-27T06:01:52.946 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.946 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 25: conflicting requests 2024-02-27T06:01:52.946 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.946 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 26: conflicting requests 2024-02-27T06:01:52.946 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.946 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 27: conflicting requests 2024-02-27T06:01:52.946 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.947 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 28: conflicting requests 2024-02-27T06:01:52.947 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.947 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 29: conflicting requests 2024-02-27T06:01:52.947 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.947 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 30: conflicting requests 2024-02-27T06:01:52.947 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.947 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 31: conflicting requests 2024-02-27T06:01:52.947 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.947 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 32: conflicting requests 2024-02-27T06:01:52.947 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.947 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 33: conflicting requests 2024-02-27T06:01:52.947 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.948 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 34: conflicting requests 2024-02-27T06:01:52.948 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.948 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 35: conflicting requests 2024-02-27T06:01:52.948 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240203221207:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.948 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 36: conflicting requests 2024-02-27T06:01:52.948 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.948 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.948 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.948 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 37: cannot install the best candidate for the job 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 38: conflicting requests 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 39: cannot install the best candidate for the job 2024-02-27T06:01:52.949 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.950 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.950 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.950 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.950 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 40: cannot install the best candidate for the job 2024-02-27T06:01:52.950 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.950 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.950 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.950 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.950 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 41: cannot install the best candidate for the job 2024-02-27T06:01:52.950 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 42: cannot install the best candidate for the job 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 43: cannot install the best candidate for the job 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: Problem 44: cannot install the best candidate for the job 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.951 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-27T06:01:52.952 INFO:teuthology.orchestra.run.smithi131.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-27T06:01:53.355 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-27T06:01:53.355 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-27T06:01:53.461 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-02-27T06:01:53.470 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-02-27T06:01:53.470 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2024-02-27T06:01:53.470 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-02-27T06:01:53.471 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-02-27T06:01:53.471 INFO:teuthology.orchestra.run.smithi131.stdout: kernel x86_64 5.14.0-419.el9 baseos 6.0 M 2024-02-27T06:01:53.471 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-02-27T06:01:53.471 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-core x86_64 5.14.0-419.el9 baseos 21 M 2024-02-27T06:01:53.471 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules x86_64 5.14.0-419.el9 baseos 39 M 2024-02-27T06:01:53.471 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-core x86_64 5.14.0-419.el9 baseos 33 M 2024-02-27T06:01:53.471 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:01:53.471 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-02-27T06:01:53.471 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-02-27T06:01:53.471 INFO:teuthology.orchestra.run.smithi131.stdout:Install 4 Packages 2024-02-27T06:01:53.471 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:01:53.472 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 99 M 2024-02-27T06:01:53.472 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 122 M 2024-02-27T06:01:53.472 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-02-27T06:01:54.091 INFO:teuthology.orchestra.run.smithi131.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 12 MB/s | 6.0 MB 00:00 2024-02-27T06:01:54.633 INFO:teuthology.orchestra.run.smithi131.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 20 MB/s | 21 MB 00:01 2024-02-27T06:01:55.233 INFO:teuthology.orchestra.run.smithi131.stdout:(3/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 24 MB/s | 39 MB 00:01 2024-02-27T06:01:55.834 INFO:teuthology.orchestra.run.smithi131.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 19 MB/s | 33 MB 00:01 2024-02-27T06:01:55.834 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-02-27T06:01:55.834 INFO:teuthology.orchestra.run.smithi131.stdout:Total 42 MB/s | 99 MB 00:02 2024-02-27T06:01:56.721 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-02-27T06:01:57.101 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-02-27T06:01:57.101 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-02-27T06:01:58.737 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-02-27T06:01:58.737 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-02-27T06:02:01.418 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-02-27T06:02:02.068 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-27T06:02:02.196 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-27T06:02:04.931 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-27T06:02:05.109 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-27T06:02:09.830 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-27T06:02:10.064 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-27T06:02:14.777 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-27T06:02:19.260 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 313 kB/s | 19 MB 01:02 2024-02-27T06:02:24.335 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.1 MB 00:01 2024-02-27T06:02:26.169 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2024-02-27T06:02:28.546 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 9.2 MB/s | 20 MB 00:02 2024-02-27T06:02:33.513 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-02-27T06:02:35.779 INFO:teuthology.orchestra.run.smithi150.stderr:Modular dependency problems: 2024-02-27T06:02:35.779 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-02-27T06:02:35.779 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 1: conflicting requests 2024-02-27T06:02:35.779 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240219154654:20125149.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.779 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 2: conflicting requests 2024-02-27T06:02:35.779 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.779 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 3: conflicting requests 2024-02-27T06:02:35.779 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.779 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 4: conflicting requests 2024-02-27T06:02:35.779 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 5: conflicting requests 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 6: conflicting requests 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 7: conflicting requests 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 8: conflicting requests 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 9: conflicting requests 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 10: conflicting requests 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.780 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 11: conflicting requests 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 12: conflicting requests 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 13: conflicting requests 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 14: conflicting requests 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 15: conflicting requests 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 16: conflicting requests 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 17: conflicting requests 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 18: conflicting requests 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.781 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 19: conflicting requests 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 20: conflicting requests 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 21: conflicting requests 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 22: conflicting requests 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 23: conflicting requests 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 24: conflicting requests 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 25: conflicting requests 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 26: conflicting requests 2024-02-27T06:02:35.782 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 27: conflicting requests 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 28: conflicting requests 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 29: conflicting requests 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 30: conflicting requests 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 31: conflicting requests 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 32: conflicting requests 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 33: conflicting requests 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.783 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 34: conflicting requests 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 35: conflicting requests 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240203221207:9edba152.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 36: conflicting requests 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 37: cannot install the best candidate for the job 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 38: conflicting requests 2024-02-27T06:02:35.784 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 39: cannot install the best candidate for the job 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 40: cannot install the best candidate for the job 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.785 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 41: cannot install the best candidate for the job 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 42: cannot install the best candidate for the job 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 43: cannot install the best candidate for the job 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.786 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.787 INFO:teuthology.orchestra.run.smithi150.stderr: Problem 44: cannot install the best candidate for the job 2024-02-27T06:02:35.787 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.787 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.787 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-27T06:02:35.787 INFO:teuthology.orchestra.run.smithi150.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-27T06:02:36.204 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-27T06:02:36.204 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-27T06:02:36.313 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-02-27T06:02:36.320 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-27T06:02:36.320 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-02-27T06:02:36.320 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-27T06:02:36.320 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-02-27T06:02:36.321 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-419.el9 baseos 6.0 M 2024-02-27T06:02:36.321 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-02-27T06:02:36.321 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core x86_64 5.14.0-419.el9 baseos 21 M 2024-02-27T06:02:36.321 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules x86_64 5.14.0-419.el9 baseos 39 M 2024-02-27T06:02:36.321 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core x86_64 5.14.0-419.el9 baseos 33 M 2024-02-27T06:02:36.321 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:02:36.321 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-02-27T06:02:36.321 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-02-27T06:02:36.321 INFO:teuthology.orchestra.run.smithi150.stdout:Install 4 Packages 2024-02-27T06:02:36.321 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:02:36.321 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 99 M 2024-02-27T06:02:36.321 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 122 M 2024-02-27T06:02:36.322 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-02-27T06:02:43.633 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-27T06:02:43.633 INFO:teuthology.orchestra.run.smithi131.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-27T06:02:43.633 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:02:43.715 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-27T06:02:44.170 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-27T06:02:44.170 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-27T06:02:44.170 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-27T06:02:44.170 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-27T06:02:44.602 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-27T06:02:44.602 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:02:44.602 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-02-27T06:02:44.603 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-27T06:02:44.603 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-27T06:02:44.603 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-27T06:02:44.603 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-27T06:02:44.603 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:02:44.603 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-02-27T06:02:45.104 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-02-27T06:02:45.265 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-27T06:02:45.266 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-27T06:02:45.266 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-27T06:02:45.266 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-27T06:02:45.266 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-27T06:02:45.266 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-27T06:02:45.266 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi131.front.sepia.ceph.com, path=None, version=distro) 2024-02-27T06:02:45.266 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-02-27T06:02:45.387 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-27T06:02:45.388 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-27T06:02:45.388 DEBUG:teuthology.orchestra.run.smithi131:> sudo rpm -qi grub2-tools 2024-02-27T06:02:45.425 INFO:teuthology.orchestra.run.smithi131.stdout:Name : grub2-tools 2024-02-27T06:02:45.425 INFO:teuthology.orchestra.run.smithi131.stdout:Epoch : 1 2024-02-27T06:02:45.425 INFO:teuthology.orchestra.run.smithi131.stdout:Version : 2.06 2024-02-27T06:02:45.425 INFO:teuthology.orchestra.run.smithi131.stdout:Release : 68.el9 2024-02-27T06:02:45.425 INFO:teuthology.orchestra.run.smithi131.stdout:Architecture: x86_64 2024-02-27T06:02:45.425 INFO:teuthology.orchestra.run.smithi131.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:Group : Unspecified 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:Size : 8264254 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:License : GPLv3+ 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:Packager : builder@centos.org 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:Vendor : CentOS 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:Summary : Support tools for GRUB. 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:Description : 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-27T06:02:45.426 INFO:teuthology.orchestra.run.smithi131.stdout:hardware devices. 2024-02-27T06:02:45.427 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:02:45.427 INFO:teuthology.orchestra.run.smithi131.stdout:This subpackage provides tools for support of all platforms. 2024-02-27T06:02:45.427 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-27T06:02:45.427 INFO:teuthology.task.kernel:Updating grub on smithi131 to boot 5.14.0-419.el9.x86_64 2024-02-27T06:02:45.428 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-27T06:02:46.059 INFO:teuthology.orchestra.run.smithi131.stderr:Generating grub configuration file ... 2024-02-27T06:02:47.484 INFO:teuthology.orchestra.run.smithi131.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-27T06:02:47.536 INFO:teuthology.orchestra.run.smithi131.stderr:done 2024-02-27T06:02:47.538 DEBUG:teuthology.orchestra.run.smithi131:> mktemp 2024-02-27T06:02:47.557 INFO:teuthology.orchestra.run.smithi131.stdout:/tmp/tmp.IISQEUaKak 2024-02-27T06:02:47.557 DEBUG:teuthology.orchestra.run.smithi131:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IISQEUaKak 2024-02-27T06:02:47.627 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0666 /tmp/tmp.IISQEUaKak 2024-02-27T06:02:47.761 DEBUG:teuthology.orchestra.remote:smithi131:/tmp/tmp.IISQEUaKak is 6KB 2024-02-27T06:02:47.810 DEBUG:teuthology.orchestra.run.smithi131:> rm -fr /tmp/tmp.IISQEUaKak 2024-02-27T06:02:47.824 DEBUG:teuthology.orchestra.run.smithi131:> sudo /bin/ls /boot/loader/entries || true 2024-02-27T06:02:47.888 INFO:teuthology.orchestra.run.smithi131.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-27T06:02:47.888 INFO:teuthology.orchestra.run.smithi131.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-27T06:02:47.888 INFO:teuthology.orchestra.run.smithi131.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-27T06:02:47.888 INFO:teuthology.orchestra.run.smithi131.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-27T06:02:47.893 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-27T06:02:48.038 DEBUG:teuthology.orchestra.run.smithi131:> sudo shutdown -r now 2024-02-27T06:02:54.772 INFO:teuthology.orchestra.run.smithi150.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 339 kB/s | 6.0 MB 00:18 2024-02-27T06:03:18.041 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-27T06:03:18.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-02-27T06:03:18.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T06:04:01.152 INFO:teuthology.orchestra.run.smithi150.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 252 kB/s | 21 MB 01:24 2024-02-27T06:04:18.044 DEBUG:teuthology.orchestra.remote:timed out 2024-02-27T06:04:27.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-02-27T06:04:27.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T06:04:27.472 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-02-27T06:04:27.887 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-02-27T06:04:27.887 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-419.el9.x86_64"... 2024-02-27T06:04:27.887 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-02-27T06:04:27.901 INFO:teuthology.orchestra.run.smithi131.stdout:5.14.0-419.el9.x86_64 2024-02-27T06:04:27.901 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-27T06:04:27.902 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-27T06:04:27.902 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-27T06:04:28.902 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-27T06:04:28.902 DEBUG:teuthology.orchestra.run.smithi131:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-27T06:04:28.981 INFO:teuthology.orchestra.run.smithi131.stdout:ttyS1 2024-02-27T06:04:29.025 DEBUG:teuthology.parallel:result is None 2024-02-27T06:05:04.057 INFO:teuthology.orchestra.run.smithi150.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 264 kB/s | 33 MB 02:09 2024-02-27T06:05:12.382 INFO:teuthology.orchestra.run.smithi150.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 257 kB/s | 39 MB 02:35 2024-02-27T06:05:12.382 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-02-27T06:05:12.382 INFO:teuthology.orchestra.run.smithi150.stdout:Total 652 kB/s | 99 MB 02:36 2024-02-27T06:05:13.260 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-02-27T06:05:13.654 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-02-27T06:05:13.654 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-02-27T06:05:15.317 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-02-27T06:05:15.318 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-02-27T06:05:18.236 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-02-27T06:05:18.897 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-27T06:05:19.024 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-27T06:05:21.802 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-27T06:05:21.984 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-27T06:05:26.697 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-27T06:05:26.930 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-27T06:05:31.648 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-27T06:06:00.666 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-27T06:06:00.667 INFO:teuthology.orchestra.run.smithi150.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-27T06:06:00.667 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:06:00.752 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-27T06:06:01.272 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-27T06:06:01.273 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-27T06:06:01.273 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-27T06:06:01.273 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-27T06:06:01.743 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-27T06:06:01.743 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:06:01.743 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-02-27T06:06:01.743 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-27T06:06:01.743 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-27T06:06:01.743 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-27T06:06:01.743 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-27T06:06:01.743 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:06:01.743 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-02-27T06:06:02.354 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-02-27T06:06:02.499 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-27T06:06:02.499 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-27T06:06:02.499 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-27T06:06:02.499 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-27T06:06:02.499 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-27T06:06:02.499 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-27T06:06:02.499 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2024-02-27T06:06:02.500 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-02-27T06:06:02.621 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-27T06:06:02.622 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-27T06:06:02.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -qi grub2-tools 2024-02-27T06:06:02.664 INFO:teuthology.orchestra.run.smithi150.stdout:Name : grub2-tools 2024-02-27T06:06:02.664 INFO:teuthology.orchestra.run.smithi150.stdout:Epoch : 1 2024-02-27T06:06:02.664 INFO:teuthology.orchestra.run.smithi150.stdout:Version : 2.06 2024-02-27T06:06:02.664 INFO:teuthology.orchestra.run.smithi150.stdout:Release : 68.el9 2024-02-27T06:06:02.664 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: x86_64 2024-02-27T06:06:02.664 INFO:teuthology.orchestra.run.smithi150.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-27T06:06:02.664 INFO:teuthology.orchestra.run.smithi150.stdout:Group : Unspecified 2024-02-27T06:06:02.664 INFO:teuthology.orchestra.run.smithi150.stdout:Size : 8264254 2024-02-27T06:06:02.664 INFO:teuthology.orchestra.run.smithi150.stdout:License : GPLv3+ 2024-02-27T06:06:02.664 INFO:teuthology.orchestra.run.smithi150.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-27T06:06:02.664 INFO:teuthology.orchestra.run.smithi150.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-27T06:06:02.664 INFO:teuthology.orchestra.run.smithi150.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout:Packager : builder@centos.org 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout:Vendor : CentOS 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout:Summary : Support tools for GRUB. 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout:Description : 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout:hardware devices. 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:06:02.665 INFO:teuthology.orchestra.run.smithi150.stdout:This subpackage provides tools for support of all platforms. 2024-02-27T06:06:02.666 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-27T06:06:02.666 INFO:teuthology.task.kernel:Updating grub on smithi150 to boot 5.14.0-419.el9.x86_64 2024-02-27T06:06:02.666 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-27T06:06:03.266 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2024-02-27T06:06:04.612 INFO:teuthology.orchestra.run.smithi150.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-27T06:06:04.644 INFO:teuthology.orchestra.run.smithi150.stderr:done 2024-02-27T06:06:04.647 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-02-27T06:06:04.662 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.bqHOitAksQ 2024-02-27T06:06:04.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bqHOitAksQ 2024-02-27T06:06:04.730 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.bqHOitAksQ 2024-02-27T06:06:04.871 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.bqHOitAksQ is 6KB 2024-02-27T06:06:04.920 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.bqHOitAksQ 2024-02-27T06:06:04.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo /bin/ls /boot/loader/entries || true 2024-02-27T06:06:04.999 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-27T06:06:04.999 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-27T06:06:04.999 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-27T06:06:04.999 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-27T06:06:05.000 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-27T06:06:05.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2024-02-27T06:06:35.147 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-27T06:06:35.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-27T06:06:35.148 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T06:07:35.150 DEBUG:teuthology.orchestra.remote:timed out 2024-02-27T06:07:44.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-27T06:07:44.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T06:08:00.121 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-02-27T06:08:00.637 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-27T06:08:00.637 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-419.el9.x86_64"... 2024-02-27T06:08:00.638 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-02-27T06:08:00.661 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-419.el9.x86_64 2024-02-27T06:08:00.661 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-27T06:08:00.661 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-27T06:08:00.661 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-27T06:08:01.662 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-27T06:08:01.662 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-27T06:08:01.753 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2024-02-27T06:08:01.808 DEBUG:teuthology.parallel:result is None 2024-02-27T06:08:01.808 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-27T06:08:01.817 INFO:teuthology.task.internal:Creating test directory... 2024-02-27T06:08:01.817 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-27T06:08:01.820 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-27T06:08:01.836 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-27T06:08:01.857 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-27T06:08:01.898 INFO:teuthology.task.internal:Creating archive directory... 2024-02-27T06:08:01.899 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-27T06:08:01.901 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-27T06:08:02.145 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-27T06:08:02.154 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-27T06:08:02.154 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-27T06:08:02.156 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-27T06:08:02.190 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-27T06:08:02.193 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-27T06:08:02.202 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-27T06:08:02.207 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-27T06:08:02.209 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-27T06:08:02.216 INFO:teuthology.task.internal:Configuring sudo... 2024-02-27T06:08:02.216 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-27T06:08:02.251 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-27T06:08:02.278 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-27T06:08:02.286 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-27T06:08:02.286 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-27T06:08:02.320 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-27T06:08:02.335 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-27T06:08:02.417 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-27T06:08:02.503 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-27T06:08:02.543 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-27T06:08:02.587 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-02-27T06:08:02.587 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-27T06:08:02.654 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-27T06:08:02.697 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-27T06:08:02.749 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-27T06:08:02.828 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-27T06:08:02.873 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-27T06:08:02.873 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-27T06:08:02.939 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2024-02-27T06:08:02.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-02-27T06:08:02.978 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-27T06:08:03.032 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-27T06:08:03.538 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-27T06:08:03.546 INFO:teuthology.task.internal:Starting timer... 2024-02-27T06:08:03.546 INFO:teuthology.run_tasks:Running task pcp... 2024-02-27T06:08:03.594 INFO:teuthology.run_tasks:Running task selinux... 2024-02-27T06:08:03.633 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-27T06:08:03.634 DEBUG:teuthology.orchestra.run.smithi131:> sudo service auditd rotate 2024-02-27T06:08:03.701 INFO:teuthology.orchestra.run.smithi131.stdout:Rotating logs: 2024-02-27T06:08:03.703 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2024-02-27T06:08:03.751 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2024-02-27T06:08:03.753 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-27T06:08:03.754 DEBUG:teuthology.orchestra.run.smithi131:> /usr/sbin/getenforce 2024-02-27T06:08:03.785 INFO:teuthology.orchestra.run.smithi131.stdout:Permissive 2024-02-27T06:08:03.786 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2024-02-27T06:08:03.808 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2024-02-27T06:08:03.809 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi131.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive'} 2024-02-27T06:08:03.809 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-27T06:08:03.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T06:08:03.854 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-27T06:08:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T06:08:03.882 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-27T06:08:03.882 DEBUG:teuthology.orchestra.run.smithi131:> sudo /usr/sbin/setenforce permissive 2024-02-27T06:08:03.929 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2024-02-27T06:08:03.978 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-27T06:08:03.992 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-27T06:08:03.996 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-27T06:08:04.114 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-27T06:08:04.128 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-27T06:08:04.129 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi131.front.sepia.ceph.com,smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-27T06:12:55.612 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2024-02-27T06:12:55.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-02-27T06:12:55.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T06:12:55.705 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-02-27T06:12:55.783 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-02-27T06:12:55.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-27T06:12:55.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-27T06:12:55.871 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-02-27T06:12:55.959 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-02-27T06:12:55.959 INFO:teuthology.run_tasks:Running task clock... 2024-02-27T06:12:55.970 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-27T06:12:55.970 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-27T06:12:55.971 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-27T06:12:55.973 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-27T06:12:55.973 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-27T06:12:56.002 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-27T06:12:56.018 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-27T06:12:56.051 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: ntpd: command not found 2024-02-27T06:12:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-27T06:12:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-27T06:12:56.061 INFO:teuthology.orchestra.run.smithi131.stdout:506 Cannot talk to daemon 2024-02-27T06:12:56.075 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2024-02-27T06:12:56.077 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-27T06:12:56.089 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2024-02-27T06:12:56.094 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-27T06:12:56.105 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-27T06:12:56.121 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-27T06:12:56.146 INFO:teuthology.orchestra.run.smithi131.stderr:bash: line 1: ntpq: command not found 2024-02-27T06:12:56.150 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-27T06:12:56.150 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2024-02-27T06:12:56.150 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T06:12:56.150 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T06:12:56.150 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T06:12:56.150 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T06:12:56.173 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-02-27T06:12:56.176 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-27T06:12:56.176 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-02-27T06:12:56.176 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T06:12:56.176 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T06:12:56.177 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T06:12:56.177 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T06:12:56.178 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-27T06:12:56.273 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'} 2024-02-27T06:12:56.273 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-02-27T06:12:56.274 INFO:tasks.cephadm:Cluster fsid is 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:12:56.274 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-27T06:12:56.274 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-27T06:12:56.275 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi131': '172.21.15.131', 'mon.smithi150': '172.21.15.150'} 2024-02-27T06:12:56.275 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-27T06:12:56.275 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname $(hostname -s) 2024-02-27T06:12:56.302 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname $(hostname -s) 2024-02-27T06:12:56.330 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-02-27T06:12:56.330 DEBUG:teuthology.orchestra.run.smithi131:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-27T06:12:56.727 INFO:teuthology.orchestra.run.smithi131.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 27 06:12 /home/ubuntu/cephtest/cephadm 2024-02-27T06:12:56.728 DEBUG:teuthology.orchestra.run.smithi150:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-27T06:12:56.903 INFO:teuthology.orchestra.run.smithi150.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 27 06:12 /home/ubuntu/cephtest/cephadm 2024-02-27T06:12:56.904 DEBUG:teuthology.orchestra.run.smithi131:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-27T06:12:56.920 DEBUG:teuthology.orchestra.run.smithi150:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-27T06:12:56.944 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-02-27T06:12:56.944 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-27T06:12:56.962 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-27T06:12:57.821 INFO:teuthology.orchestra.run.smithi150.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-27T06:12:57.833 INFO:teuthology.orchestra.run.smithi131.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-27T06:13:21.472 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:13:21.472 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-27T06:13:21.472 INFO:teuthology.orchestra.run.smithi131.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-27T06:13:21.472 INFO:teuthology.orchestra.run.smithi131.stdout: "repo_digests": [ 2024-02-27T06:13:21.472 INFO:teuthology.orchestra.run.smithi131.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-27T06:13:21.472 INFO:teuthology.orchestra.run.smithi131.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-27T06:13:21.472 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-02-27T06:13:21.472 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:13:21.508 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-02-27T06:13:21.509 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-27T06:13:21.509 INFO:teuthology.orchestra.run.smithi150.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-27T06:13:21.509 INFO:teuthology.orchestra.run.smithi150.stdout: "repo_digests": [ 2024-02-27T06:13:21.509 INFO:teuthology.orchestra.run.smithi150.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-27T06:13:21.509 INFO:teuthology.orchestra.run.smithi150.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-27T06:13:21.510 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-02-27T06:13:21.510 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-02-27T06:13:21.529 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /etc/ceph 2024-02-27T06:13:21.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph 2024-02-27T06:13:21.591 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 777 /etc/ceph 2024-02-27T06:13:21.626 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /etc/ceph 2024-02-27T06:13:21.659 INFO:tasks.cephadm:Writing seed config... 2024-02-27T06:13:21.660 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-27T06:13:21.660 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-27T06:13:21.660 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-27T06:13:21.660 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-27T06:13:21.660 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-27T06:13:21.660 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-27T06:13:21.661 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-27T06:13:21.661 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-27T06:13:21.661 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-02-27T06:13:21.661 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-27T06:13:21.681 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3fa4c766-d537-11ee-95c3-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-27T06:13:21.681 DEBUG:teuthology.orchestra.run.smithi131:mon.smithi131> sudo journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi131.service 2024-02-27T06:13:21.723 INFO:tasks.cephadm:Bootstrapping... 2024-02-27T06:13:21.724 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.131 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-27T06:13:21.918 INFO:teuthology.orchestra.run.smithi131.stderr:-------------------------------------------------------------------------------- 2024-02-27T06:13:21.918 INFO:teuthology.orchestra.run.smithi131.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '3fa4c766-d537-11ee-95c3-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.131', '--skip-admin-label'] 2024-02-27T06:13:21.944 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: 4.9.3-dev 2024-02-27T06:13:21.944 INFO:teuthology.orchestra.run.smithi131.stderr:Verifying podman|docker is present... 2024-02-27T06:13:21.965 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: 4.9.3-dev 2024-02-27T06:13:21.967 INFO:teuthology.orchestra.run.smithi131.stderr:Verifying lvm2 is present... 2024-02-27T06:13:21.968 INFO:teuthology.orchestra.run.smithi131.stderr:Verifying time synchronization is in place... 2024-02-27T06:13:21.974 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-27T06:13:21.980 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: inactive 2024-02-27T06:13:21.987 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: enabled 2024-02-27T06:13:21.993 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: active 2024-02-27T06:13:21.993 INFO:teuthology.orchestra.run.smithi131.stderr:Unit chronyd.service is enabled and running 2024-02-27T06:13:21.993 INFO:teuthology.orchestra.run.smithi131.stderr:Repeating the final host check... 2024-02-27T06:13:22.015 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: 4.9.3-dev 2024-02-27T06:13:22.017 INFO:teuthology.orchestra.run.smithi131.stderr:podman (/bin/podman) version 4.9.3 is present 2024-02-27T06:13:22.017 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl is present 2024-02-27T06:13:22.017 INFO:teuthology.orchestra.run.smithi131.stderr:lvcreate is present 2024-02-27T06:13:22.023 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-27T06:13:22.029 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: inactive 2024-02-27T06:13:22.035 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: enabled 2024-02-27T06:13:22.041 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: active 2024-02-27T06:13:22.042 INFO:teuthology.orchestra.run.smithi131.stderr:Unit chronyd.service is enabled and running 2024-02-27T06:13:22.042 INFO:teuthology.orchestra.run.smithi131.stderr:Host looks OK 2024-02-27T06:13:22.042 INFO:teuthology.orchestra.run.smithi131.stderr:Cluster fsid: 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:13:22.042 INFO:teuthology.orchestra.run.smithi131.stderr:Acquiring lock 139752248901008 on /run/cephadm/3fa4c766-d537-11ee-95c3-87774f69a715.lock 2024-02-27T06:13:22.042 INFO:teuthology.orchestra.run.smithi131.stderr:Lock 139752248901008 acquired on /run/cephadm/3fa4c766-d537-11ee-95c3-87774f69a715.lock 2024-02-27T06:13:22.042 INFO:teuthology.orchestra.run.smithi131.stderr:Verifying IP 172.21.15.131 port 3300 ... 2024-02-27T06:13:22.043 INFO:teuthology.orchestra.run.smithi131.stderr:Verifying IP 172.21.15.131 port 6789 ... 2024-02-27T06:13:22.043 INFO:teuthology.orchestra.run.smithi131.stderr:Base mon IP is 172.21.15.131, final addrv is [v2:172.21.15.131:3300,v1:172.21.15.131:6789] 2024-02-27T06:13:22.045 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.131 metric 100 2024-02-27T06:13:22.045 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.131 metric 100 2024-02-27T06:13:22.048 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-27T06:13:22.048 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-27T06:13:22.048 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1789sec hoplimit 64 pref medium 2024-02-27T06:13:22.050 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-27T06:13:22.051 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-27T06:13:22.051 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-27T06:13:22.051 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-27T06:13:22.051 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:72f9/64 scope link 2024-02-27T06:13:22.051 INFO:teuthology.orchestra.run.smithi131.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-27T06:13:22.051 INFO:teuthology.orchestra.run.smithi131.stderr:Mon IP `172.21.15.131` is in CIDR network `172.21.0.0/20` 2024-02-27T06:13:22.052 INFO:teuthology.orchestra.run.smithi131.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-27T06:13:22.052 INFO:teuthology.orchestra.run.smithi131.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-27T06:13:22.152 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-02-27T06:13:22.549 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Getting image source signatures 2024-02-27T06:13:22.549 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-02-27T06:13:22.549 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-02-27T06:13:22.549 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-02-27T06:13:22.549 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-02-27T06:13:22.549 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-02-27T06:13:22.549 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-27T06:13:22.716 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: Writing manifest to image destination 2024-02-27T06:13:22.877 INFO:teuthology.orchestra.run.smithi131.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-27T06:13:23.268 INFO:teuthology.orchestra.run.smithi131.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-27T06:13:23.810 INFO:teuthology.orchestra.run.smithi131.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-27T06:13:23.810 INFO:teuthology.orchestra.run.smithi131.stderr:Extracting ceph user uid/gid from container image... 2024-02-27T06:13:27.471 INFO:teuthology.orchestra.run.smithi131.stderr:stat: 167 167 2024-02-27T06:13:33.261 INFO:teuthology.orchestra.run.smithi131.stderr:Creating initial keys... 2024-02-27T06:13:35.608 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-authtool: AQCPfd1l3PmOHxAA+kWw7vIRwPoCEi4ziUQ3cA== 2024-02-27T06:13:36.480 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-authtool: AQCQfd1ldPHTGRAAO9qJUosu3KZYG3ESuy/Q6w== 2024-02-27T06:13:37.444 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph-authtool: AQCRfd1lcFSfFRAApN71+k04QDw8e2Qii+IQFQ== 2024-02-27T06:13:37.942 INFO:teuthology.orchestra.run.smithi131.stderr:Creating initial monmap... 2024-02-27T06:13:38.358 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-27T06:13:38.358 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-02-27T06:13:38.358 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:13:38.358 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-27T06:13:39.574 INFO:teuthology.orchestra.run.smithi131.stderr:monmaptool for smithi131 [v2:172.21.15.131:3300,v1:172.21.15.131:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-27T06:13:39.574 INFO:teuthology.orchestra.run.smithi131.stderr:setting min_mon_release = octopus 2024-02-27T06:13:39.574 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/monmaptool: set fsid to 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:13:39.574 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-27T06:13:39.574 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-02-27T06:13:39.574 INFO:teuthology.orchestra.run.smithi131.stderr:Creating mon... 2024-02-27T06:13:40.529 INFO:teuthology.orchestra.run.smithi131.stderr:create mon.smithi131 on 2024-02-27T06:13:40.733 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-27T06:13:41.022 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3fa4c766-d537-11ee-95c3-87774f69a715.target → /etc/systemd/system/ceph-3fa4c766-d537-11ee-95c3-87774f69a715.target. 2024-02-27T06:13:41.022 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-3fa4c766-d537-11ee-95c3-87774f69a715.target → /etc/systemd/system/ceph-3fa4c766-d537-11ee-95c3-87774f69a715.target. 2024-02-27T06:13:41.575 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Failed to reset failed state of unit ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi131.service: Unit ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi131.service not loaded. 2024-02-27T06:13:41.584 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3fa4c766-d537-11ee-95c3-87774f69a715.target.wants/ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi131.service → /etc/systemd/system/ceph-3fa4c766-d537-11ee-95c3-87774f69a715@.service. 2024-02-27T06:13:42.253 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:42 smithi131 podman[24470]: 2024-02-27 06:13:42.186530042 +0000 UTC m=+0.314754561 container start 5d0dd24c717440ccb2c62d637b63d14065c3cfd4a750cf300a68e86b6c86f729 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, io.buildah.version=1.19.8, RELEASE=HEAD, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., architecture=x86_64, version=8, GIT_BRANCH=HEAD) 2024-02-27T06:13:42.296 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: disabled 2024-02-27T06:13:42.302 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: inactive 2024-02-27T06:13:42.303 INFO:teuthology.orchestra.run.smithi131.stderr:firewalld.service is not enabled 2024-02-27T06:13:42.303 INFO:teuthology.orchestra.run.smithi131.stderr:Not possible to enable service . firewalld.service is not available 2024-02-27T06:13:42.304 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for mon to start... 2024-02-27T06:13:42.304 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for mon... 2024-02-27T06:13:42.509 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:42 smithi131 ceph-mon[24489]: mkfs 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:13:42.509 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:42 smithi131 ceph-mon[24489]: mon.smithi131 is new leader, mons smithi131 in quorum (ranks 0) 2024-02-27T06:13:42.509 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:42 smithi131 bash[24470]: 5d0dd24c717440ccb2c62d637b63d14065c3cfd4a750cf300a68e86b6c86f729 2024-02-27T06:13:42.509 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:42 smithi131 systemd[1]: Started Ceph mon.smithi131 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:13:42.772 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: cluster: 2024-02-27T06:13:42.772 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: id: 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:13:42.772 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-27T06:13:42.772 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2024-02-27T06:13:42.772 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: services: 2024-02-27T06:13:42.772 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi131 (age 0.52712s) 2024-02-27T06:13:42.773 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-27T06:13:42.773 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-27T06:13:42.773 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2024-02-27T06:13:42.773 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: data: 2024-02-27T06:13:42.773 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-27T06:13:42.773 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-27T06:13:42.773 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-27T06:13:42.773 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: pgs: 2024-02-27T06:13:42.773 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2024-02-27T06:13:43.342 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:43 smithi131 ceph-mon[24489]: mon.smithi131 is new leader, mons smithi131 in quorum (ranks 0) 2024-02-27T06:13:43.342 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:43 smithi131 ceph-mon[24489]: monmap e1: 1 mons at {smithi131=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} 2024-02-27T06:13:43.342 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:43 smithi131 ceph-mon[24489]: fsmap 2024-02-27T06:13:43.342 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:43 smithi131 ceph-mon[24489]: osdmap e1: 0 total, 0 up, 0 in 2024-02-27T06:13:43.342 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:43 smithi131 ceph-mon[24489]: mgrmap e1: no daemons active 2024-02-27T06:13:43.342 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:43 smithi131 ceph-mon[24489]: from='client.? 172.21.15.131:0/1966910633' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-27T06:13:43.374 INFO:teuthology.orchestra.run.smithi131.stderr:mon is available 2024-02-27T06:13:43.374 INFO:teuthology.orchestra.run.smithi131.stderr:Assimilating anything we can from ceph.conf... 2024-02-27T06:13:43.868 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2024-02-27T06:13:43.868 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: [global] 2024-02-27T06:13:43.868 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: fsid = 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:13:43.868 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.131:3300,v1:172.21.15.131:6789] 2024-02-27T06:13:43.869 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-27T06:13:43.869 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-27T06:13:43.869 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-27T06:13:43.869 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-27T06:13:43.869 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2024-02-27T06:13:43.869 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: [mgr] 2024-02-27T06:13:43.869 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-27T06:13:43.869 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2024-02-27T06:13:43.869 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: [osd] 2024-02-27T06:13:43.869 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-27T06:13:43.869 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-27T06:13:44.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:44 smithi131 ceph-mon[24489]: from='client.? 172.21.15.131:0/1801395462' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-27T06:13:44.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:44 smithi131 ceph-mon[24489]: from='client.? 172.21.15.131:0/1801395462' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-27T06:13:44.420 INFO:teuthology.orchestra.run.smithi131.stderr:Generating new minimal ceph.conf... 2024-02-27T06:13:45.483 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:45 smithi131 ceph-mon[24489]: from='client.? 172.21.15.131:0/3907718328' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:13:45.486 INFO:teuthology.orchestra.run.smithi131.stderr:Restarting the monitor... 2024-02-27T06:13:45.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:45 smithi131 systemd[1]: Stopping Ceph mon.smithi131 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:13:45.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:45 smithi131 bash[24708]: Error: no container with name or ID "ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon.smithi131" found: no such container 2024-02-27T06:13:45.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:45 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131[24485]: 2024-02-27T06:13:45.647+0000 7f2341fc6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi131 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:13:45.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:45 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131[24485]: 2024-02-27T06:13:45.647+0000 7f2341fc6700 -1 mon.smithi131@0(leader) e1 *** Got Signal Terminated *** 2024-02-27T06:13:46.279 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:45 smithi131 podman[24721]: 2024-02-27 06:13:45.893369405 +0000 UTC m=+0.349753457 container died 5d0dd24c717440ccb2c62d637b63d14065c3cfd4a750cf300a68e86b6c86f729 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, ceph=True, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, RELEASE=HEAD, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , release=754, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-02-27T06:13:46.279 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 podman[24721]: 2024-02-27 06:13:46.131183187 +0000 UTC m=+0.587567234 container cleanup 5d0dd24c717440ccb2c62d637b63d14065c3cfd4a750cf300a68e86b6c86f729 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, distribution-scope=public, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, ceph=True, io.k8s.display-name=CentOS Stream 8, vcs-type=git, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_CLEAN=True) 2024-02-27T06:13:46.279 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 bash[24721]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131 2024-02-27T06:13:46.554 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 podman[24735]: 2024-02-27 06:13:46.282374133 +0000 UTC m=+0.388897580 container remove 5d0dd24c717440ccb2c62d637b63d14065c3cfd4a750cf300a68e86b6c86f729 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, io.openshift.expose-services=, io.buildah.version=1.19.8, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, ceph=True, architecture=x86_64, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc.) 2024-02-27T06:13:46.554 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 bash[24747]: Error: no container with name or ID "ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon.smithi131" found: no such container 2024-02-27T06:13:46.555 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi131.service: Deactivated successfully. 2024-02-27T06:13:46.555 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 systemd[1]: Stopped Ceph mon.smithi131 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:13:46.555 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 systemd[1]: Starting Ceph mon.smithi131 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:13:46.832 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 podman[24815]: 2024-02-27 06:13:46.558043879 +0000 UTC m=+0.114241500 container create 8e3c593dcb9f7fcf100b2019e24ea771949230f151d3723b649cf3ac9eed94f4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_BRANCH=HEAD, release=754, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, distribution-scope=public, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.openshift.expose-services=, io.buildah.version=1.19.8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream) 2024-02-27T06:13:46.832 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 podman[24815]: 2024-02-27 06:13:46.461872905 +0000 UTC m=+0.018070535 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-02-27T06:13:46.832 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 podman[24815]: 2024-02-27 06:13:46.760457948 +0000 UTC m=+0.316655581 container init 8e3c593dcb9f7fcf100b2019e24ea771949230f151d3723b649cf3ac9eed94f4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, version=8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, release=754, com.redhat.component=centos-stream-container) 2024-02-27T06:13:46.832 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 podman[24815]: 2024-02-27 06:13:46.763237891 +0000 UTC m=+0.319435512 container start 8e3c593dcb9f7fcf100b2019e24ea771949230f151d3723b649cf3ac9eed94f4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, architecture=x86_64, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vendor=Red Hat, Inc., vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, RELEASE=HEAD, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-27T06:13:46.832 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: set uid:gid to 167:167 (ceph:ceph) 2024-02-27T06:13:46.832 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-27T06:13:46.832 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: pidfile_write: ignore empty --pid-file 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: load: jerasure load: lrc 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: RocksDB version: 6.15.5 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Compile date Apr 18 2022 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: DB SUMMARY 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: DB Session ID: EFFVKSY6GHRP3NBME9ML 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: CURRENT file: CURRENT 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: IDENTITY file: IDENTITY 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi131/store.db dir, Total Num: 1, files: 000008.sst 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi131/store.db: 000010.log size: 82741 ; 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.error_if_exists: 0 2024-02-27T06:13:46.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.create_if_missing: 0 2024-02-27T06:13:46.834 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.paranoid_checks: 1 2024-02-27T06:13:46.834 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-27T06:13:46.834 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.env: 0x55e5712c6860 2024-02-27T06:13:46.834 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.fs: Posix File System 2024-02-27T06:13:46.834 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.info_log: 0x55e5720c1fa0 2024-02-27T06:13:46.834 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_file_opening_threads: 16 2024-02-27T06:13:46.834 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.statistics: (nil) 2024-02-27T06:13:46.834 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.use_fsync: 0 2024-02-27T06:13:46.834 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_log_file_size: 0 2024-02-27T06:13:46.834 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-27T06:13:46.835 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-27T06:13:46.836 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.keep_log_file_num: 1000 2024-02-27T06:13:46.836 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.recycle_log_file_num: 0 2024-02-27T06:13:46.836 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.allow_fallocate: 1 2024-02-27T06:13:46.836 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.allow_mmap_reads: 0 2024-02-27T06:13:46.836 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.allow_mmap_writes: 0 2024-02-27T06:13:46.836 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.use_direct_reads: 0 2024-02-27T06:13:46.836 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-27T06:13:46.836 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.create_missing_column_families: 0 2024-02-27T06:13:46.836 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.db_log_dir: 2024-02-27T06:13:46.836 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi131/store.db 2024-02-27T06:13:46.836 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-27T06:13:46.836 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.advise_random_on_open: 1 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.db_write_buffer_size: 0 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.write_buffer_manager: 0x55e5721b22d0 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.rate_limiter: (nil) 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-27T06:13:46.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.wal_recovery_mode: 2 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.enable_thread_tracking: 0 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.enable_pipelined_write: 0 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.unordered_write: 0 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.row_cache: None 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.wal_filter: None 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.allow_ingest_behind: 0 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.preserve_deletes: 0 2024-02-27T06:13:46.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.two_write_queues: 0 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.manual_wal_flush: 0 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.atomic_flush: 0 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.log_readahead_size: 0 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.best_efforts_recovery: 0 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.allow_data_in_errors: 0 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.db_host_id: __hostname__ 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_background_jobs: 2 2024-02-27T06:13:46.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_background_compactions: -1 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_subcompactions: 1 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_total_wal_size: 0 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_open_files: -1 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.bytes_per_sync: 0 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-27T06:13:46.840 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-27T06:13:46.841 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_readahead_size: 0 2024-02-27T06:13:46.841 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_background_flushes: -1 2024-02-27T06:13:46.841 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Compression algorithms supported: 2024-02-27T06:13:46.841 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-27T06:13:46.841 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: kZSTD supported: 0 2024-02-27T06:13:46.841 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: kXpressCompression supported: 0 2024-02-27T06:13:46.841 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: kLZ4HCCompression supported: 1 2024-02-27T06:13:46.841 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: kLZ4Compression supported: 1 2024-02-27T06:13:46.841 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: kBZip2Compression supported: 0 2024-02-27T06:13:46.841 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: kZlibCompression supported: 1 2024-02-27T06:13:46.841 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: kSnappyCompression supported: 1 2024-02-27T06:13:46.841 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi131/store.db/MANIFEST-000009 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.merge_operator: 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_filter: None 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_filter_factory: None 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.sst_partitioner_factory: None 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e57208dd00) 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: cache_index_and_filter_blocks: 1 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-27T06:13:46.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: pin_top_level_index_and_filter: 1 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: index_type: 0 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: data_block_index_type: 0 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: index_shortening: 1 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: hash_index_allow_collision: 1 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: checksum: 1 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: no_block_cache: 0 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_cache: 0x55e5720f8170 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_cache_name: BinnedLRUCache 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_cache_options: 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: capacity : 536870912 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: num_shard_bits : 4 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: strict_capacity_limit : 0 2024-02-27T06:13:46.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: high_pri_pool_ratio: 0.000 2024-02-27T06:13:46.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_cache_compressed: (nil) 2024-02-27T06:13:46.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: persistent_cache: (nil) 2024-02-27T06:13:46.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_size: 4096 2024-02-27T06:13:46.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_size_deviation: 10 2024-02-27T06:13:46.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_restart_interval: 16 2024-02-27T06:13:46.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: index_block_restart_interval: 1 2024-02-27T06:13:46.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: metadata_block_size: 4096 2024-02-27T06:13:46.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: partition_filters: 0 2024-02-27T06:13:46.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: use_delta_encoding: 1 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: whole_key_filtering: 1 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: verify_compression: 0 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: read_amp_bytes_per_bit: 0 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: format_version: 4 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: enable_index_compression: 1 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_align: 0 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.write_buffer_size: 33554432 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_write_buffer_number: 2 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compression: NoCompression 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.bottommost_compression: Disabled 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.prefix_extractor: nullptr 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-27T06:13:46.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.num_levels: 7 2024-02-27T06:13:46.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-27T06:13:46.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-27T06:13:46.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-27T06:13:46.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-27T06:13:46.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-27T06:13:46.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-27T06:13:46.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-27T06:13:46.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-27T06:13:46.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-27T06:13:46.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-27T06:13:46.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-27T06:13:46.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compression_opts.level: 32767 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compression_opts.strategy: 0 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compression_opts.enabled: false 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.target_file_size_base: 67108864 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-27T06:13:46.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-27T06:13:46.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-27T06:13:46.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-27T06:13:46.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-27T06:13:46.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-27T06:13:46.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-27T06:13:46.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-27T06:13:46.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-27T06:13:46.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.arena_block_size: 4194304 2024-02-27T06:13:46.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-27T06:13:46.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-27T06:13:46.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-27T06:13:46.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.disable_auto_compactions: 0 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.table_properties_collectors: 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.inplace_update_support: 0 2024-02-27T06:13:46.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-27T06:13:46.850 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-27T06:13:46.850 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-27T06:13:46.850 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-27T06:13:46.850 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.bloom_locality: 0 2024-02-27T06:13:46.850 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.max_successive_merges: 0 2024-02-27T06:13:46.850 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-27T06:13:46.850 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.paranoid_file_checks: 0 2024-02-27T06:13:46.850 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.force_consistency_checks: 1 2024-02-27T06:13:46.850 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.report_bg_io_stats: 0 2024-02-27T06:13:46.850 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.ttl: 2592000 2024-02-27T06:13:46.850 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-27T06:13:46.850 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.enable_blob_files: false 2024-02-27T06:13:46.851 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.min_blob_size: 0 2024-02-27T06:13:46.851 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.blob_file_size: 268435456 2024-02-27T06:13:46.851 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-27T06:13:46.851 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-27T06:13:46.851 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-27T06:13:46.851 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi131/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-27T06:13:46.851 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-02-27T06:13:46.851 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-02-27T06:13:46.851 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709014426788240, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-27T06:13:46.851 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-02-27T06:13:46.851 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-27T06:13:46.851 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709014426788926, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79423, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77687, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71996, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1709014426, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "f83a36e6-548c-4317-bc8a-6137ab43e9d0", "db_session_id": "EFFVKSY6GHRP3NBME9ML"}} 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709014426789256, "job": 1, "event": "recovery_finished"} 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi131/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55e5720dea80 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: DB pointer 0x55e5720ee000 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ** DB Stats ** 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T06:13:46.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-27T06:13:46.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-27T06:13:46.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T06:13:46.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:13:46.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ** Compaction Stats [default] ** 2024-02-27T06:13:46.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T06:13:46.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T06:13:46.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: L0 2/0 79.36 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.4 0.00 0.00 1 0.000 0 0 2024-02-27T06:13:46.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Sum 2/0 79.36 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.4 0.00 0.00 1 0.000 0 0 2024-02-27T06:13:46.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.4 0.00 0.00 1 0.000 0 0 2024-02-27T06:13:46.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:13:46.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ** Compaction Stats [default] ** 2024-02-27T06:13:46.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 165.4 0.00 0.00 1 0.000 0 0 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Cumulative compaction: 0.00 GB write, 26.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Interval compaction: 0.00 GB write, 26.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T06:13:46.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:13:46.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ** Compaction Stats [default] ** 2024-02-27T06:13:46.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T06:13:46.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T06:13:46.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: L0 2/0 79.36 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.4 0.00 0.00 1 0.000 0 0 2024-02-27T06:13:46.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Sum 2/0 79.36 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.4 0.00 0.00 1 0.000 0 0 2024-02-27T06:13:46.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-27T06:13:46.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:13:46.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ** Compaction Stats [default] ** 2024-02-27T06:13:46.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T06:13:46.855 INFO:teuthology.orchestra.run.smithi131.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-27T06:13:46.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T06:13:46.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 165.4 0.00 0.00 1 0.000 0 0 2024-02-27T06:13:46.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T06:13:46.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-27T06:13:46.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T06:13:46.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T06:13:46.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T06:13:46.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Cumulative compaction: 0.00 GB write, 26.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: starting mon.smithi131 rank 0 at public addrs [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] at bind addrs [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi131 fsid 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: mon.smithi131@-1(???) e1 preinit fsid 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: mon.smithi131@-1(???).mds e1 new map 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: mon.smithi131@-1(???).mds e1 print_map 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: e1 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-27T06:13:46.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: legacy client fscid: -1 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: No filesystems configured 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: mon.smithi131@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: mon.smithi131@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: mon.smithi131@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: mon.smithi131@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: mon.smithi131@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: expand_channel_meta expand map: {default=false} 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: expand_channel_meta from 'false' to 'false' 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: expand_channel_meta expanded map: {default=false} 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: expand_channel_meta expand map: {default=info} 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: expand_channel_meta from 'info' to 'info' 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: expand_channel_meta expanded map: {default=info} 2024-02-27T06:13:46.858 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: expand_channel_meta expand map: {default=daemon} 2024-02-27T06:13:46.859 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: expand_channel_meta from 'daemon' to 'daemon' 2024-02-27T06:13:46.859 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: expand_channel_meta expanded map: {default=daemon} 2024-02-27T06:13:46.859 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: expand_channel_meta expand map: {default=debug} 2024-02-27T06:13:46.859 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: expand_channel_meta from 'debug' to 'debug' 2024-02-27T06:13:46.859 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: expand_channel_meta expanded map: {default=debug} 2024-02-27T06:13:46.859 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 bash[24815]: 8e3c593dcb9f7fcf100b2019e24ea771949230f151d3723b649cf3ac9eed94f4 2024-02-27T06:13:46.859 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 systemd[1]: Started Ceph mon.smithi131 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:13:47.092 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: mon.smithi131 is new leader, mons smithi131 in quorum (ranks 0) 2024-02-27T06:13:47.092 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: monmap e1: 1 mons at {smithi131=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} 2024-02-27T06:13:47.092 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: fsmap 2024-02-27T06:13:47.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: osdmap e1: 0 total, 0 up, 0 in 2024-02-27T06:13:47.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:46 smithi131 ceph-mon[24834]: mgrmap e1: no daemons active 2024-02-27T06:13:47.925 INFO:teuthology.orchestra.run.smithi131.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-27T06:13:47.925 INFO:teuthology.orchestra.run.smithi131.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-27T06:13:47.925 INFO:teuthology.orchestra.run.smithi131.stderr:Creating mgr... 2024-02-27T06:13:47.926 INFO:teuthology.orchestra.run.smithi131.stderr:Verifying port 9283 ... 2024-02-27T06:13:48.092 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Failed to reset failed state of unit ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mgr.smithi131.frtgqo.service: Unit ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mgr.smithi131.frtgqo.service not loaded. 2024-02-27T06:13:48.099 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3fa4c766-d537-11ee-95c3-87774f69a715.target.wants/ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mgr.smithi131.frtgqo.service → /etc/systemd/system/ceph-3fa4c766-d537-11ee-95c3-87774f69a715@.service. 2024-02-27T06:13:48.700 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:48 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/3238404760' entity='client.admin' 2024-02-27T06:13:48.814 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: disabled 2024-02-27T06:13:48.821 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: inactive 2024-02-27T06:13:48.821 INFO:teuthology.orchestra.run.smithi131.stderr:firewalld.service is not enabled 2024-02-27T06:13:48.822 INFO:teuthology.orchestra.run.smithi131.stderr:Not possible to enable service . firewalld.service is not available 2024-02-27T06:13:48.830 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: disabled 2024-02-27T06:13:48.837 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: inactive 2024-02-27T06:13:48.838 INFO:teuthology.orchestra.run.smithi131.stderr:firewalld.service is not enabled 2024-02-27T06:13:48.838 INFO:teuthology.orchestra.run.smithi131.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-27T06:13:48.838 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for mgr to start... 2024-02-27T06:13:48.838 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for mgr... 2024-02-27T06:13:49.415 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2024-02-27T06:13:49.415 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "fsid": "3fa4c766-d537-11ee-95c3-87774f69a715", 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "health": { 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "checks": {}, 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mutes": [] 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum": [ 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 0 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "smithi131" 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2024-02-27T06:13:49.416 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-27T06:13:49.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "monmap": { 2024-02-27T06:13:49.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T06:13:49.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-27T06:13:49.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-27T06:13:49.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:49.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osdmap": { 2024-02-27T06:13:49.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T06:13:49.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-27T06:13:49.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-27T06:13:49.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-27T06:13:49.417 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "pgmap": { 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-27T06:13:49.421 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "fsmap": { 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "available": false, 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "modules": [ 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "iostat", 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "nfs", 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "restful" 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "services": {} 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:49.422 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "servicemap": { 2024-02-27T06:13:49.423 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T06:13:49.423 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "modified": "2024-02-27T06:13:42.248112+0000", 2024-02-27T06:13:49.423 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "services": {} 2024-02-27T06:13:49.423 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:49.423 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-27T06:13:49.423 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2024-02-27T06:13:49.592 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:49 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2622969896' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-27T06:13:50.461 INFO:teuthology.orchestra.run.smithi131.stderr:mgr not available, waiting (1/15)... 2024-02-27T06:13:53.022 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2024-02-27T06:13:53.022 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2024-02-27T06:13:53.022 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "fsid": "3fa4c766-d537-11ee-95c3-87774f69a715", 2024-02-27T06:13:53.022 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "health": { 2024-02-27T06:13:53.022 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-27T06:13:53.022 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "checks": {}, 2024-02-27T06:13:53.022 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mutes": [] 2024-02-27T06:13:53.022 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:53.022 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum": [ 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 0 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "smithi131" 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "monmap": { 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osdmap": { 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-27T06:13:53.023 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-27T06:13:53.024 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-27T06:13:53.024 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-27T06:13:53.027 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-27T06:13:53.027 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-27T06:13:53.027 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:53.027 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "pgmap": { 2024-02-27T06:13:53.027 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-27T06:13:53.027 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-27T06:13:53.027 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "fsmap": { 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "available": false, 2024-02-27T06:13:53.028 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "modules": [ 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "iostat", 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "nfs", 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "restful" 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "services": {} 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "servicemap": { 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "modified": "2024-02-27T06:13:42.248112+0000", 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "services": {} 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-27T06:13:53.029 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2024-02-27T06:13:53.143 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:53 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2614983177' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-27T06:13:53.626 INFO:teuthology.orchestra.run.smithi131.stderr:mgr not available, waiting (2/15)... 2024-02-27T06:13:55.092 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: Activating manager daemon smithi131.frtgqo 2024-02-27T06:13:55.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: mgrmap e2: smithi131.frtgqo(active, starting, since 0.00292857s) 2024-02-27T06:13:55.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: from='mgr.14100 172.21.15.131:0/271745850' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T06:13:55.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: from='mgr.14100 172.21.15.131:0/271745850' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T06:13:55.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: from='mgr.14100 172.21.15.131:0/271745850' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T06:13:55.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: from='mgr.14100 172.21.15.131:0/271745850' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:13:55.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: from='mgr.14100 172.21.15.131:0/271745850' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi131.frtgqo", "id": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:13:55.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: Manager daemon smithi131.frtgqo is now available 2024-02-27T06:13:55.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: from='mgr.14100 172.21.15.131:0/271745850' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:13:55.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: from='mgr.14100 172.21.15.131:0/271745850' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:13:55.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: from='mgr.14100 172.21.15.131:0/271745850' entity='mgr.smithi131.frtgqo' 2024-02-27T06:13:55.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: from='mgr.14100 172.21.15.131:0/271745850' entity='mgr.smithi131.frtgqo' 2024-02-27T06:13:55.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:54 smithi131 ceph-mon[24834]: from='mgr.14100 172.21.15.131:0/271745850' entity='mgr.smithi131.frtgqo' 2024-02-27T06:13:56.295 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2024-02-27T06:13:56.295 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2024-02-27T06:13:56.295 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "fsid": "3fa4c766-d537-11ee-95c3-87774f69a715", 2024-02-27T06:13:56.296 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "health": { 2024-02-27T06:13:56.296 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-27T06:13:56.296 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "checks": {}, 2024-02-27T06:13:56.296 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mutes": [] 2024-02-27T06:13:56.296 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:56.296 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-27T06:13:56.296 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum": [ 2024-02-27T06:13:56.296 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 0 2024-02-27T06:13:56.296 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2024-02-27T06:13:56.296 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-27T06:13:56.296 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "smithi131" 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "monmap": { 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osdmap": { 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-27T06:13:56.297 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "pgmap": { 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-27T06:13:56.301 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "fsmap": { 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "available": true, 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "modules": [ 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "iostat", 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "nfs", 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "restful" 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ], 2024-02-27T06:13:56.302 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "services": {} 2024-02-27T06:13:56.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:56.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "servicemap": { 2024-02-27T06:13:56.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-27T06:13:56.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "modified": "2024-02-27T06:13:42.248112+0000", 2024-02-27T06:13:56.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "services": {} 2024-02-27T06:13:56.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: }, 2024-02-27T06:13:56.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-27T06:13:56.303 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2024-02-27T06:13:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:56 smithi131 ceph-mon[24834]: mgrmap e3: smithi131.frtgqo(active, since 1.00653s) 2024-02-27T06:13:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:56 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/3427493111' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-27T06:13:56.906 INFO:teuthology.orchestra.run.smithi131.stderr:mgr is available 2024-02-27T06:13:57.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2024-02-27T06:13:57.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: [global] 2024-02-27T06:13:57.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: fsid = 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:13:57.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-27T06:13:57.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-27T06:13:57.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-27T06:13:57.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-27T06:13:57.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2024-02-27T06:13:57.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: [mgr] 2024-02-27T06:13:57.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-27T06:13:57.454 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 2024-02-27T06:13:57.455 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: [osd] 2024-02-27T06:13:57.455 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-27T06:13:57.455 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-27T06:13:58.029 INFO:teuthology.orchestra.run.smithi131.stderr:Enabling cephadm module... 2024-02-27T06:13:58.092 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:57 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2612850078' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-27T06:13:58.092 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:57 smithi131 ceph-mon[24834]: mgrmap e4: smithi131.frtgqo(active, since 3s) 2024-02-27T06:13:59.092 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:58 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1339520243' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-27T06:13:59.995 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:59 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1339520243' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-27T06:13:59.995 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:13:59 smithi131 ceph-mon[24834]: mgrmap e5: smithi131.frtgqo(active, since 4s) 2024-02-27T06:14:00.087 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2024-02-27T06:14:00.087 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-27T06:14:00.087 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "available": true, 2024-02-27T06:14:00.087 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "active_name": "smithi131.frtgqo", 2024-02-27T06:14:00.087 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-27T06:14:00.087 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2024-02-27T06:14:00.643 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for the mgr to restart... 2024-02-27T06:14:00.643 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for mgr epoch 5... 2024-02-27T06:14:00.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:00 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/4168368548' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-27T06:14:05.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:04 smithi131 ceph-mon[24834]: Active manager daemon smithi131.frtgqo restarted 2024-02-27T06:14:05.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:04 smithi131 ceph-mon[24834]: Activating manager daemon smithi131.frtgqo 2024-02-27T06:14:05.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:04 smithi131 ceph-mon[24834]: osdmap e2: 0 total, 0 up, 0 in 2024-02-27T06:14:05.733 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2024-02-27T06:14:05.733 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-27T06:14:05.733 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "initialized": true 2024-02-27T06:14:05.733 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2024-02-27T06:14:05.973 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: mgrmap e6: smithi131.frtgqo(active, starting, since 0.0525396s) 2024-02-27T06:14:05.973 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:14:05.973 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi131.frtgqo", "id": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:14:05.973 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T06:14:05.973 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T06:14:05.973 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T06:14:05.973 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: Manager daemon smithi131.frtgqo is now available 2024-02-27T06:14:05.973 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:05.973 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:05.974 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:14:05.974 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:14:05.974 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:14:05.974 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:14:05.974 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:05 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:14:06.208 INFO:teuthology.orchestra.run.smithi131.stderr:mgr epoch 5 is available 2024-02-27T06:14:06.208 INFO:teuthology.orchestra.run.smithi131.stderr:Setting orchestrator backend to cephadm... 2024-02-27T06:14:06.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:06 smithi131 ceph-mon[24834]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-27T06:14:06.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:06 smithi131 ceph-mon[24834]: mgrmap e7: smithi131.frtgqo(active, since 1.0595s) 2024-02-27T06:14:06.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:06 smithi131 ceph-mon[24834]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-27T06:14:06.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:06 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:06.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:06 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:14:07.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:07 smithi131 ceph-mon[24834]: [27/Feb/2024:06:14:05] ENGINE Bus STARTING 2024-02-27T06:14:07.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:07 smithi131 ceph-mon[24834]: [27/Feb/2024:06:14:06] ENGINE Serving on https://172.21.15.131:7150 2024-02-27T06:14:07.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:07 smithi131 ceph-mon[24834]: [27/Feb/2024:06:14:06] ENGINE Bus STARTED 2024-02-27T06:14:07.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:07 smithi131 ceph-mon[24834]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:07.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:07 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:07.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:07 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:14:07.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:07 smithi131 ceph-mon[24834]: mgrmap e8: smithi131.frtgqo(active, since 2s) 2024-02-27T06:14:07.912 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: value unchanged 2024-02-27T06:14:08.503 INFO:teuthology.orchestra.run.smithi131.stderr:Generating ssh key... 2024-02-27T06:14:09.340 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:08 smithi131 ceph-mon[24834]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:10.566 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:10 smithi131 ceph-mon[24834]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:10.567 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:10 smithi131 ceph-mon[24834]: Generating ssh key... 2024-02-27T06:14:10.567 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:10 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:10.567 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:10 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:10.567 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeG3aL4/j4t1O1YQdmopDE3EoJMtUr3G1P+NQr1VKv/6Ca2mnOCsnkT8JPa8g2xWY898ew7jR+Mep20zlSufH1KZ22t+sR/9RtOPy1L3bbzx8Cet7rpiwdMgCdXEDbmg7ZQqTiBqEffoQJ2B8wYxmHN08sMUssXXLY7BFgb7aPj7UA/6EXbQ9cUbhJK7Fy9+kdY0pZYhbbgCQ748Byl+d+nv5x3xzmteL/E1uLI3WA/8EqkwtYCVPuaoCz954xdn2FruxyB9j8px4yXl5Bjp1pMZr/1PdpXxHXxritSP04yRIhiMZ61f3JbuH/O+uQDMyZwmp+7/I/4xiaScBmvudqfV5HBQY2nAzTWR1vD+Rg4MN0WZY7llxNCLK3dcpUK6KD4r09Tmm7al5vkTu8XOi5EZYmB6mi03SNRWeErKkjnfPTRz3DZIzPiywfB3ecGktw4uVIdsNan5iqZYtCmwz6o4p/n83Zs1LhXACa+kpLJPtPydnYhZkO/Rw6yMfrEXU= ceph-3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:14:11.167 INFO:teuthology.orchestra.run.smithi131.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-27T06:14:11.167 INFO:teuthology.orchestra.run.smithi131.stderr:Adding key to root@localhost authorized_keys... 2024-02-27T06:14:11.168 INFO:teuthology.orchestra.run.smithi131.stderr:Adding host smithi131... 2024-02-27T06:14:11.381 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:11 smithi131 ceph-mon[24834]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:12.760 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Added host 'smithi131' with addr '172.21.15.131' 2024-02-27T06:14:13.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:12 smithi131 ceph-mon[24834]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi131", "addr": "172.21.15.131", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:13.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:12 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:13.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:12 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:14:13.341 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying mon service with default placement... 2024-02-27T06:14:13.971 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-27T06:14:13.983 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:13 smithi131 ceph-mon[24834]: Deploying cephadm binary to smithi131 2024-02-27T06:14:13.983 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:13 smithi131 ceph-mon[24834]: Added host smithi131 2024-02-27T06:14:14.775 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying mgr service with default placement... 2024-02-27T06:14:15.003 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:14 smithi131 ceph-mon[24834]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:15.003 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:14 smithi131 ceph-mon[24834]: Saving service mon spec with placement count:5 2024-02-27T06:14:15.004 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:14 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:15.518 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-27T06:14:16.786 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:16 smithi131 ceph-mon[24834]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:16.786 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:16 smithi131 ceph-mon[24834]: Saving service mgr spec with placement count:2 2024-02-27T06:14:16.786 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:16 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:16.786 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:16 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:16.788 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying crash service with default placement... 2024-02-27T06:14:17.376 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-27T06:14:18.020 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying prometheus service with default placement... 2024-02-27T06:14:18.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:18 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:18.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:18 smithi131 ceph-mon[24834]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:18.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:18 smithi131 ceph-mon[24834]: Saving service crash spec with placement * 2024-02-27T06:14:18.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:18 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:18.624 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-27T06:14:19.286 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying grafana service with default placement... 2024-02-27T06:14:19.879 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:19 smithi131 ceph-mon[24834]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:19.880 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:19 smithi131 ceph-mon[24834]: Saving service prometheus spec with placement count:1 2024-02-27T06:14:19.880 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:19 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:20.286 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-27T06:14:20.820 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying node-exporter service with default placement... 2024-02-27T06:14:20.935 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:20 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:20.936 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:20 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:20.936 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:20 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:21.430 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-27T06:14:21.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:21 smithi131 ceph-mon[24834]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:21.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:21 smithi131 ceph-mon[24834]: Saving service grafana spec with placement count:1 2024-02-27T06:14:21.843 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:21 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:23.014 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:22 smithi131 ceph-mon[24834]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:23.015 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:22 smithi131 ceph-mon[24834]: Saving service node-exporter spec with placement * 2024-02-27T06:14:23.018 INFO:teuthology.orchestra.run.smithi131.stderr:Deploying alertmanager service with default placement... 2024-02-27T06:14:23.661 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-27T06:14:24.835 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:24 smithi131 ceph-mon[24834]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:24.835 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:24 smithi131 ceph-mon[24834]: Saving service alertmanager spec with placement count:1 2024-02-27T06:14:24.835 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:24 smithi131 ceph-mon[24834]: from='mgr.14118 172.21.15.131:0/1098758960' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:26.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:25 smithi131 ceph-mon[24834]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:14:26.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:25 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1427269207' entity='client.admin' 2024-02-27T06:14:26.486 INFO:teuthology.orchestra.run.smithi131.stderr:Enabling the dashboard module... 2024-02-27T06:14:27.343 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:26 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/3789589514' entity='client.admin' 2024-02-27T06:14:27.343 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:26 smithi131 ceph-mon[24834]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:14:28.232 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:27 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/3996839773' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-27T06:14:29.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:28 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/3996839773' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-27T06:14:29.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:28 smithi131 ceph-mon[24834]: mgrmap e9: smithi131.frtgqo(active, since 23s) 2024-02-27T06:14:29.210 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2024-02-27T06:14:29.210 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-27T06:14:29.210 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "available": true, 2024-02-27T06:14:29.210 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "active_name": "smithi131.frtgqo", 2024-02-27T06:14:29.210 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-27T06:14:29.210 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2024-02-27T06:14:29.918 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for the mgr to restart... 2024-02-27T06:14:29.919 INFO:teuthology.orchestra.run.smithi131.stderr:Waiting for mgr epoch 9... 2024-02-27T06:14:30.002 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:29 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1827057095' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-27T06:14:34.915 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: { 2024-02-27T06:14:34.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: Active manager daemon smithi131.frtgqo restarted 2024-02-27T06:14:34.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: Activating manager daemon smithi131.frtgqo 2024-02-27T06:14:34.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: osdmap e3: 0 total, 0 up, 0 in 2024-02-27T06:14:34.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: mgrmap e10: smithi131.frtgqo(active, starting, since 0.0536103s) 2024-02-27T06:14:34.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:14:34.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi131.frtgqo", "id": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:14:34.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T06:14:34.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T06:14:34.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T06:14:34.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: Manager daemon smithi131.frtgqo is now available 2024-02-27T06:14:34.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:14:34.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:14:34.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:14:34.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:14:34.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:34 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:34.916 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-27T06:14:34.916 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: "initialized": true 2024-02-27T06:14:34.917 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: } 2024-02-27T06:14:35.637 INFO:teuthology.orchestra.run.smithi131.stderr:mgr epoch 9 is available 2024-02-27T06:14:35.637 INFO:teuthology.orchestra.run.smithi131.stderr:Generating a dashboard self-signed certificate... 2024-02-27T06:14:36.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:35 smithi131 ceph-mon[24834]: [27/Feb/2024:06:14:34] ENGINE Bus STARTING 2024-02-27T06:14:36.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:35 smithi131 ceph-mon[24834]: [27/Feb/2024:06:14:34] ENGINE Serving on https://172.21.15.131:7150 2024-02-27T06:14:36.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:35 smithi131 ceph-mon[24834]: [27/Feb/2024:06:14:34] ENGINE Bus STARTED 2024-02-27T06:14:36.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:35 smithi131 ceph-mon[24834]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-27T06:14:36.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:35 smithi131 ceph-mon[24834]: mgrmap e11: smithi131.frtgqo(active, since 1.05572s) 2024-02-27T06:14:36.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:35 smithi131 ceph-mon[24834]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-27T06:14:36.273 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:35 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:36.527 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-27T06:14:37.145 INFO:teuthology.orchestra.run.smithi131.stderr:Creating initial admin user... 2024-02-27T06:14:37.395 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:37 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:37.395 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:37 smithi131 ceph-mon[24834]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:37.395 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:37 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:37.395 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:37 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:37.395 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:37 smithi131 ceph-mon[24834]: mgrmap e12: smithi131.frtgqo(active, since 2s) 2024-02-27T06:14:38.033 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$u/5LEKisUZoMyC1iPr.T.OBpcVqrpHx9xLTuZKm10B0P/d/2mMzLG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1709014478, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-27T06:14:38.343 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:38 smithi131 ceph-mon[24834]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:38.343 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:38 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:38.805 INFO:teuthology.orchestra.run.smithi131.stderr:Fetching dashboard port number... 2024-02-27T06:14:39.439 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: 8443 2024-02-27T06:14:39.802 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:39 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1177423067' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-27T06:14:40.342 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: disabled 2024-02-27T06:14:40.349 INFO:teuthology.orchestra.run.smithi131.stderr:systemctl: inactive 2024-02-27T06:14:40.350 INFO:teuthology.orchestra.run.smithi131.stderr:firewalld.service is not enabled 2024-02-27T06:14:40.350 INFO:teuthology.orchestra.run.smithi131.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-27T06:14:40.352 INFO:teuthology.orchestra.run.smithi131.stderr:Ceph Dashboard is now available at: 2024-02-27T06:14:40.352 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-02-27T06:14:40.352 INFO:teuthology.orchestra.run.smithi131.stderr: URL: https://smithi131.front.sepia.ceph.com:8443/ 2024-02-27T06:14:40.352 INFO:teuthology.orchestra.run.smithi131.stderr: User: admin 2024-02-27T06:14:40.352 INFO:teuthology.orchestra.run.smithi131.stderr: Password: qk25o97rqm 2024-02-27T06:14:40.352 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-02-27T06:14:40.353 INFO:teuthology.orchestra.run.smithi131.stderr:Enabling autotune for osd_memory_target 2024-02-27T06:14:41.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:41 smithi131 ceph-mon[24834]: mgrmap e13: smithi131.frtgqo(active, since 6s) 2024-02-27T06:14:42.289 INFO:teuthology.orchestra.run.smithi131.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-27T06:14:42.888 INFO:teuthology.orchestra.run.smithi131.stderr:You can access the Ceph CLI with: 2024-02-27T06:14:42.888 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-02-27T06:14:42.888 INFO:teuthology.orchestra.run.smithi131.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-27T06:14:42.888 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-02-27T06:14:42.888 INFO:teuthology.orchestra.run.smithi131.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-27T06:14:42.888 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-02-27T06:14:42.888 INFO:teuthology.orchestra.run.smithi131.stderr: ceph telemetry on 2024-02-27T06:14:42.888 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-02-27T06:14:42.888 INFO:teuthology.orchestra.run.smithi131.stderr:For more information see: 2024-02-27T06:14:42.888 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-02-27T06:14:42.889 INFO:teuthology.orchestra.run.smithi131.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-27T06:14:42.889 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-02-27T06:14:42.889 INFO:teuthology.orchestra.run.smithi131.stderr:Bootstrap complete. 2024-02-27T06:14:42.923 INFO:tasks.cephadm:Fetching config... 2024-02-27T06:14:42.923 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-02-27T06:14:42.923 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-27T06:14:42.939 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-27T06:14:42.939 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-02-27T06:14:42.939 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-27T06:14:42.993 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-27T06:14:42.993 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-02-27T06:14:42.993 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/keyring of=/dev/stdout 2024-02-27T06:14:43.061 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-27T06:14:43.061 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-02-27T06:14:43.061 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-27T06:14:43.115 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-27T06:14:43.116 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeG3aL4/j4t1O1YQdmopDE3EoJMtUr3G1P+NQr1VKv/6Ca2mnOCsnkT8JPa8g2xWY898ew7jR+Mep20zlSufH1KZ22t+sR/9RtOPy1L3bbzx8Cet7rpiwdMgCdXEDbmg7ZQqTiBqEffoQJ2B8wYxmHN08sMUssXXLY7BFgb7aPj7UA/6EXbQ9cUbhJK7Fy9+kdY0pZYhbbgCQ748Byl+d+nv5x3xzmteL/E1uLI3WA/8EqkwtYCVPuaoCz954xdn2FruxyB9j8px4yXl5Bjp1pMZr/1PdpXxHXxritSP04yRIhiMZ61f3JbuH/O+uQDMyZwmp+7/I/4xiaScBmvudqfV5HBQY2nAzTWR1vD+Rg4MN0WZY7llxNCLK3dcpUK6KD4r09Tmm7al5vkTu8XOi5EZYmB6mi03SNRWeErKkjnfPTRz3DZIzPiywfB3ecGktw4uVIdsNan5iqZYtCmwz6o4p/n83Zs1LhXACa+kpLJPtPydnYhZkO/Rw6yMfrEXU= ceph-3fa4c766-d537-11ee-95c3-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-27T06:14:43.205 INFO:teuthology.orchestra.run.smithi131.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeG3aL4/j4t1O1YQdmopDE3EoJMtUr3G1P+NQr1VKv/6Ca2mnOCsnkT8JPa8g2xWY898ew7jR+Mep20zlSufH1KZ22t+sR/9RtOPy1L3bbzx8Cet7rpiwdMgCdXEDbmg7ZQqTiBqEffoQJ2B8wYxmHN08sMUssXXLY7BFgb7aPj7UA/6EXbQ9cUbhJK7Fy9+kdY0pZYhbbgCQ748Byl+d+nv5x3xzmteL/E1uLI3WA/8EqkwtYCVPuaoCz954xdn2FruxyB9j8px4yXl5Bjp1pMZr/1PdpXxHXxritSP04yRIhiMZ61f3JbuH/O+uQDMyZwmp+7/I/4xiaScBmvudqfV5HBQY2nAzTWR1vD+Rg4MN0WZY7llxNCLK3dcpUK6KD4r09Tmm7al5vkTu8XOi5EZYmB6mi03SNRWeErKkjnfPTRz3DZIzPiywfB3ecGktw4uVIdsNan5iqZYtCmwz6o4p/n83Zs1LhXACa+kpLJPtPydnYhZkO/Rw6yMfrEXU= ceph-3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:14:43.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeG3aL4/j4t1O1YQdmopDE3EoJMtUr3G1P+NQr1VKv/6Ca2mnOCsnkT8JPa8g2xWY898ew7jR+Mep20zlSufH1KZ22t+sR/9RtOPy1L3bbzx8Cet7rpiwdMgCdXEDbmg7ZQqTiBqEffoQJ2B8wYxmHN08sMUssXXLY7BFgb7aPj7UA/6EXbQ9cUbhJK7Fy9+kdY0pZYhbbgCQ748Byl+d+nv5x3xzmteL/E1uLI3WA/8EqkwtYCVPuaoCz954xdn2FruxyB9j8px4yXl5Bjp1pMZr/1PdpXxHXxritSP04yRIhiMZ61f3JbuH/O+uQDMyZwmp+7/I/4xiaScBmvudqfV5HBQY2nAzTWR1vD+Rg4MN0WZY7llxNCLK3dcpUK6KD4r09Tmm7al5vkTu8XOi5EZYmB6mi03SNRWeErKkjnfPTRz3DZIzPiywfB3ecGktw4uVIdsNan5iqZYtCmwz6o4p/n83Zs1LhXACa+kpLJPtPydnYhZkO/Rw6yMfrEXU= ceph-3fa4c766-d537-11ee-95c3-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-27T06:14:43.265 INFO:teuthology.orchestra.run.smithi150.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeG3aL4/j4t1O1YQdmopDE3EoJMtUr3G1P+NQr1VKv/6Ca2mnOCsnkT8JPa8g2xWY898ew7jR+Mep20zlSufH1KZ22t+sR/9RtOPy1L3bbzx8Cet7rpiwdMgCdXEDbmg7ZQqTiBqEffoQJ2B8wYxmHN08sMUssXXLY7BFgb7aPj7UA/6EXbQ9cUbhJK7Fy9+kdY0pZYhbbgCQ748Byl+d+nv5x3xzmteL/E1uLI3WA/8EqkwtYCVPuaoCz954xdn2FruxyB9j8px4yXl5Bjp1pMZr/1PdpXxHXxritSP04yRIhiMZ61f3JbuH/O+uQDMyZwmp+7/I/4xiaScBmvudqfV5HBQY2nAzTWR1vD+Rg4MN0WZY7llxNCLK3dcpUK6KD4r09Tmm7al5vkTu8XOi5EZYmB6mi03SNRWeErKkjnfPTRz3DZIzPiywfB3ecGktw4uVIdsNan5iqZYtCmwz6o4p/n83Zs1LhXACa+kpLJPtPydnYhZkO/Rw6yMfrEXU= ceph-3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:14:43.280 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-27T06:14:43.591 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:43 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1046033669' entity='client.admin' 2024-02-27T06:14:46.677 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-27T06:14:46.677 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-27T06:14:47.274 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:46 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/44745039' entity='client.admin' 2024-02-27T06:14:48.518 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:48 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:48.518 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:48 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:14:48.518 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:48 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:48.518 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:48 smithi131 ceph-mon[24834]: Deploying daemon alertmanager.smithi131 on smithi131 2024-02-27T06:14:49.653 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi150 2024-02-27T06:14:49.653 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-27T06:14:49.653 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.conf 2024-02-27T06:14:49.671 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-27T06:14:49.671 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:14:49.728 INFO:tasks.cephadm:Adding host smithi150 to orchestrator... 2024-02-27T06:14:49.729 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch host add smithi150 2024-02-27T06:14:50.034 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:49 smithi131 ceph-mon[24834]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:50.034 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:49 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:54.191 INFO:teuthology.orchestra.run.smithi131.stdout:Added host 'smithi150' with addr '172.21.15.150' 2024-02-27T06:14:55.262 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:54 smithi131 ceph-mon[24834]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi150", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:14:55.262 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:54 smithi131 ceph-mon[24834]: Deploying cephadm binary to smithi150 2024-02-27T06:14:55.262 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:54 smithi131 ceph-mon[24834]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:14:55.262 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:54 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:14:55.844 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch host ls --format=json 2024-02-27T06:14:56.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:55 smithi131 ceph-mon[24834]: Added host smithi150 2024-02-27T06:14:57.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:56 smithi131 ceph-mon[24834]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:14:58.131 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:14:58.131 INFO:teuthology.orchestra.run.smithi131.stdout:[{"addr": "172.21.15.131", "hostname": "smithi131", "labels": [], "status": ""}, {"addr": "172.21.15.150", "hostname": "smithi150", "labels": [], "status": ""}] 2024-02-27T06:14:58.866 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-27T06:14:58.866 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd crush tunables default 2024-02-27T06:14:59.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:58 smithi131 ceph-mon[24834]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:00.322 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:59 smithi131 ceph-mon[24834]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:15:00.322 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:59 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:00.323 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:59 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:15:00.323 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:59 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-27T06:15:00.323 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:14:59 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:15:01.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:00 smithi131 ceph-mon[24834]: Deploying daemon crash.smithi131 on smithi131 2024-02-27T06:15:01.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:00 smithi131 ceph-mon[24834]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:01.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:00 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/783371298' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-27T06:15:01.227 INFO:teuthology.orchestra.run.smithi131.stderr:adjusted tunables profile to default 2024-02-27T06:15:02.071 INFO:tasks.cephadm:Adding mon.smithi131 on smithi131 2024-02-27T06:15:02.071 INFO:tasks.cephadm:Adding mon.smithi150 on smithi150 2024-02-27T06:15:02.071 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch apply mon '2;smithi131:172.21.15.131=smithi131;smithi150:172.21.15.150=smithi150' 2024-02-27T06:15:02.574 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:02 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/783371298' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-27T06:15:02.574 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:02 smithi131 ceph-mon[24834]: osdmap e4: 0 total, 0 up, 0 in 2024-02-27T06:15:02.574 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:02 smithi131 ceph-mon[24834]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:04.041 INFO:teuthology.orchestra.run.smithi150.stdout:Scheduled mon update... 2024-02-27T06:15:04.586 DEBUG:teuthology.orchestra.run.smithi150:mon.smithi150> sudo journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi150.service 2024-02-27T06:15:04.589 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-27T06:15:04.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T06:15:04.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:04 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:04.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:04 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:04.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:04 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:04.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:04 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T06:15:04.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:04 smithi131 ceph-mon[24834]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T06:15:04.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:04 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:04.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:04 smithi131 ceph-mon[24834]: Deploying daemon grafana.smithi131 on smithi131 2024-02-27T06:15:04.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:04 smithi131 ceph-mon[24834]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:04.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:04 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:04.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:04 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:05.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:05 smithi131 ceph-mon[24834]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi131:172.21.15.131=smithi131;smithi150:172.21.15.150=smithi150", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:15:05.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:05 smithi131 ceph-mon[24834]: Saving service mon spec with placement smithi131:172.21.15.131=smithi131;smithi150:172.21.15.150=smithi150;count:2 2024-02-27T06:15:06.583 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:15:06.583 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","modified":"2024-02-27T06:13:38.291256Z","created":"2024-02-27T06:13:38.291256Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-27T06:15:06.585 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-02-27T06:15:06.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:06 smithi131 ceph-mon[24834]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:07.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:07 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/2317385540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:08.223 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-27T06:15:08.223 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T06:15:08.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:08 smithi131 ceph-mon[24834]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:10.099 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:15:10.099 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","modified":"2024-02-27T06:13:38.291256Z","created":"2024-02-27T06:13:38.291256Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-27T06:15:10.101 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-02-27T06:15:11.344 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:10 smithi131 ceph-mon[24834]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:11.344 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:10 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/760605680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:11.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-27T06:15:11.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T06:15:13.063 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:12 smithi131 ceph-mon[24834]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:13.537 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:15:13.537 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","modified":"2024-02-27T06:13:38.291256Z","created":"2024-02-27T06:13:38.291256Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-27T06:15:13.538 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-02-27T06:15:14.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:13 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/2139969507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:15.064 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:14 smithi131 ceph-mon[24834]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:15.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-27T06:15:15.070 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T06:15:16.944 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:15:16.944 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","modified":"2024-02-27T06:13:38.291256Z","created":"2024-02-27T06:13:38.291256Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-27T06:15:16.945 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-02-27T06:15:17.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:16 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:17.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:16 smithi131 ceph-mon[24834]: Deploying daemon node-exporter.smithi131 on smithi131 2024-02-27T06:15:17.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:16 smithi131 ceph-mon[24834]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:18.219 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:17 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/3278462033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:18.509 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-27T06:15:18.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T06:15:19.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:18 smithi131 ceph-mon[24834]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:20.300 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:15:20.300 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","modified":"2024-02-27T06:13:38.291256Z","created":"2024-02-27T06:13:38.291256Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-27T06:15:20.302 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-02-27T06:15:20.344 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:19 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:20.344 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:19 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-27T06:15:20.344 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:19 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:21.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:20 smithi131 ceph-mon[24834]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-27T06:15:21.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:20 smithi131 ceph-mon[24834]: mgrmap e14: smithi131.frtgqo(active, since 46s) 2024-02-27T06:15:21.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:20 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/565879410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:21.870 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-27T06:15:21.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T06:15:23.753 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:15:23.753 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","modified":"2024-02-27T06:13:38.291256Z","created":"2024-02-27T06:13:38.291256Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-27T06:15:23.755 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-02-27T06:15:24.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:23 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/17183213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:25.355 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-27T06:15:25.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T06:15:26.091 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:25 smithi131 ceph-mon[24834]: Active manager daemon smithi131.frtgqo restarted 2024-02-27T06:15:26.091 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:25 smithi131 ceph-mon[24834]: Activating manager daemon smithi131.frtgqo 2024-02-27T06:15:26.091 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:25 smithi131 ceph-mon[24834]: osdmap e5: 0 total, 0 up, 0 in 2024-02-27T06:15:27.069 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: mgrmap e15: smithi131.frtgqo(active, starting, since 0.0531725s) 2024-02-27T06:15:27.069 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:15:27.069 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi131.frtgqo", "id": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:15:27.069 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T06:15:27.069 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T06:15:27.070 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T06:15:27.070 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: Manager daemon smithi131.frtgqo is now available 2024-02-27T06:15:27.070 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:15:27.070 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:15:27.070 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:15:27.070 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:15:27.070 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:27.070 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:15:27.070 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:15:27.070 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:27.070 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:27.132 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:15:27.132 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","modified":"2024-02-27T06:13:38.291256Z","created":"2024-02-27T06:13:38.291256Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-27T06:15:27.134 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-02-27T06:15:27.948 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:27 smithi131 ceph-mon[24834]: [27/Feb/2024:06:15:26] ENGINE Bus STARTING 2024-02-27T06:15:27.948 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:27 smithi131 ceph-mon[24834]: [27/Feb/2024:06:15:26] ENGINE Serving on https://172.21.15.131:7150 2024-02-27T06:15:27.948 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:27 smithi131 ceph-mon[24834]: [27/Feb/2024:06:15:26] ENGINE Bus STARTED 2024-02-27T06:15:27.948 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:27 smithi131 ceph-mon[24834]: mgrmap e16: smithi131.frtgqo(active, since 1.06148s) 2024-02-27T06:15:27.948 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:27 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:27.948 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:27 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/2038090813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:28.761 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-27T06:15:28.761 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T06:15:29.203 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:28 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:29.203 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:28 smithi131 ceph-mon[24834]: mgrmap e17: smithi131.frtgqo(active, since 2s) 2024-02-27T06:15:30.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:30 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:30.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:30 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:30.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:30 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:15:31.825 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:15:31.826 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","modified":"2024-02-27T06:13:38.291256Z","created":"2024-02-27T06:13:38.291256Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-27T06:15:31.828 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-02-27T06:15:31.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:31 smithi131 ceph-mon[24834]: Updating smithi131:/etc/ceph/ceph.conf 2024-02-27T06:15:31.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:31 smithi131 ceph-mon[24834]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:15:31.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:31 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:31.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:31 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:31.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:31 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:31.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:31 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:15:31.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:31 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:31.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:31 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:15:31.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:31 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-27T06:15:31.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:31 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:15:31.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:31 smithi131 ceph-mon[24834]: Deploying daemon crash.smithi150 on smithi150 2024-02-27T06:15:33.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:32 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/2192797040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:34.218 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-27T06:15:34.218 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T06:15:35.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:35 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:35.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:35 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vvgbyr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:15:35.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:35 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vvgbyr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-27T06:15:35.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:35 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:15:35.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:35 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:15:35.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:35 smithi131 ceph-mon[24834]: Deploying daemon mgr.smithi150.vvgbyr on smithi150 2024-02-27T06:15:36.066 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:15:36.066 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":1,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","modified":"2024-02-27T06:13:38.291256Z","created":"2024-02-27T06:13:38.291256Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-27T06:15:36.068 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 1 2024-02-27T06:15:37.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:37.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:36 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/837566533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:38.614 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-27T06:15:38.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph mon dump -f json 2024-02-27T06:15:39.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:39 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:39.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:39 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:15:39.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:39 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:15:39.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:39 smithi131 ceph-mon[24834]: Deploying daemon mon.smithi150 on smithi150 2024-02-27T06:15:41.275 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 systemd[1]: Starting Ceph mon.smithi150 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:15:41.596 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 podman[26763]: 2024-02-27 06:15:41.285584434 +0000 UTC m=+0.020976448 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-02-27T06:15:41.850 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 podman[26763]: 2024-02-27 06:15:41.716526345 +0000 UTC m=+0.451918333 container create 976f96599f98d82552bd913ce1b6135c7ce3b0a452a56e634e3dad6fa75559f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, release=754, io.k8s.display-name=CentOS Stream 8, ceph=True, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , distribution-scope=public, vcs-type=git, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, architecture=x86_64, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.buildah.version=1.19.8) 2024-02-27T06:15:42.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:41 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.283 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 podman[26763]: 2024-02-27 06:15:41.969125198 +0000 UTC m=+0.704517188 container init 976f96599f98d82552bd913ce1b6135c7ce3b0a452a56e634e3dad6fa75559f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, RELEASE=HEAD, GIT_BRANCH=HEAD, name=centos-stream, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream) 2024-02-27T06:15:42.283 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 podman[26763]: 2024-02-27 06:15:41.972853735 +0000 UTC m=+0.708245724 container start 976f96599f98d82552bd913ce1b6135c7ce3b0a452a56e634e3dad6fa75559f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150, io.openshift.expose-services=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-type=git, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, release=754, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8) 2024-02-27T06:15:42.283 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: set uid:gid to 167:167 (ceph:ceph) 2024-02-27T06:15:42.283 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-27T06:15:42.283 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: pidfile_write: ignore empty --pid-file 2024-02-27T06:15:42.283 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: load: jerasure load: lrc 2024-02-27T06:15:42.284 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: RocksDB version: 6.15.5 2024-02-27T06:15:42.284 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-27T06:15:42.284 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Compile date Apr 18 2022 2024-02-27T06:15:42.284 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: DB SUMMARY 2024-02-27T06:15:42.284 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: DB Session ID: LL1PN68HNKOMOB09OGQE 2024-02-27T06:15:42.284 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: CURRENT file: CURRENT 2024-02-27T06:15:42.284 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: IDENTITY file: IDENTITY 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi150/store.db dir, Total Num: 0, files: 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi150/store.db: 000004.log size: 511 ; 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.error_if_exists: 0 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.create_if_missing: 0 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.paranoid_checks: 1 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.env: 0x559b630c6860 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.fs: Posix File System 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.info_log: 0x559b63ba9fa0 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_file_opening_threads: 16 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.statistics: (nil) 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.use_fsync: 0 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_log_file_size: 0 2024-02-27T06:15:42.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.keep_log_file_num: 1000 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.recycle_log_file_num: 0 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.allow_fallocate: 1 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.allow_mmap_reads: 0 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.allow_mmap_writes: 0 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.use_direct_reads: 0 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.create_missing_column_families: 0 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.db_log_dir: 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi150/store.db 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-27T06:15:42.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.advise_random_on_open: 1 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.db_write_buffer_size: 0 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.write_buffer_manager: 0x559b63c9a2d0 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.rate_limiter: (nil) 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.wal_recovery_mode: 2 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.enable_thread_tracking: 0 2024-02-27T06:15:42.287 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.enable_pipelined_write: 0 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.unordered_write: 0 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.row_cache: None 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.wal_filter: None 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.allow_ingest_behind: 0 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.preserve_deletes: 0 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.two_write_queues: 0 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.manual_wal_flush: 0 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.atomic_flush: 0 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-27T06:15:42.288 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.log_readahead_size: 0 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.best_efforts_recovery: 0 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.allow_data_in_errors: 0 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.db_host_id: __hostname__ 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_background_jobs: 2 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_background_compactions: -1 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_subcompactions: 1 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-27T06:15:42.289 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_total_wal_size: 0 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_open_files: -1 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.bytes_per_sync: 0 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_readahead_size: 0 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_background_flushes: -1 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Compression algorithms supported: 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: kZSTD supported: 0 2024-02-27T06:15:42.290 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: kXpressCompression supported: 0 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: kLZ4HCCompression supported: 1 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: kLZ4Compression supported: 1 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: kBZip2Compression supported: 0 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: kZlibCompression supported: 1 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: kSnappyCompression supported: 1 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi150/store.db/MANIFEST-000003 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.merge_operator: 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_filter: None 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_filter_factory: None 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.sst_partitioner_factory: None 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-27T06:15:42.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559b63b75d00) 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: cache_index_and_filter_blocks: 1 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: pin_top_level_index_and_filter: 1 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: index_type: 0 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: data_block_index_type: 0 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: index_shortening: 1 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: hash_index_allow_collision: 1 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: checksum: 1 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: no_block_cache: 0 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_cache: 0x559b63be0170 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_cache_name: BinnedLRUCache 2024-02-27T06:15:42.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_cache_options: 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: capacity : 536870912 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: num_shard_bits : 4 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: strict_capacity_limit : 0 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: high_pri_pool_ratio: 0.000 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_cache_compressed: (nil) 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: persistent_cache: (nil) 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_size: 4096 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_size_deviation: 10 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_restart_interval: 16 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: index_block_restart_interval: 1 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: metadata_block_size: 4096 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: partition_filters: 0 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: use_delta_encoding: 1 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: whole_key_filtering: 1 2024-02-27T06:15:42.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: verify_compression: 0 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: read_amp_bytes_per_bit: 0 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: format_version: 4 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: enable_index_compression: 1 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_align: 0 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.write_buffer_size: 33554432 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_write_buffer_number: 2 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compression: NoCompression 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.bottommost_compression: Disabled 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.prefix_extractor: nullptr 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.num_levels: 7 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-27T06:15:42.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compression_opts.level: 32767 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compression_opts.strategy: 0 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-27T06:15:42.295 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-27T06:15:42.296 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compression_opts.enabled: false 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.target_file_size_base: 67108864 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-27T06:15:42.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.arena_block_size: 4194304 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.disable_auto_compactions: 0 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-27T06:15:42.298 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.table_properties_collectors: 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.inplace_update_support: 0 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.bloom_locality: 0 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.max_successive_merges: 0 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.paranoid_file_checks: 0 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.force_consistency_checks: 1 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.report_bg_io_stats: 0 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.ttl: 2592000 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.enable_blob_files: false 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.min_blob_size: 0 2024-02-27T06:15:42.299 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.blob_file_size: 268435456 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi150/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709014541999192, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709014541999498, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1709014541, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "71c2abac-429d-44c2-9d4f-c067b5dabd78", "db_session_id": "LL1PN68HNKOMOB09OGQE"}} 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709014541999800, "job": 1, "event": "recovery_finished"} 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:41 smithi150 ceph-mon[26806]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi150/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-27T06:15:42.300 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x559b63bc6a80 2024-02-27T06:15:42.301 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: rocksdb: DB pointer 0x559b63bd6000 2024-02-27T06:15:42.301 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-27T06:15:42.301 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-27T06:15:42.301 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ** DB Stats ** 2024-02-27T06:15:42.301 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T06:15:42.301 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-27T06:15:42.301 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T06:15:42.301 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T06:15:42.301 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-27T06:15:42.301 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-27T06:15:42.301 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ** Compaction Stats [default] ** 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ** Compaction Stats [default] ** 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-27T06:15:42.302 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ** Compaction Stats [default] ** 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-27T06:15:42.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ** Compaction Stats [default] ** 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150 does not exist in monmap, will attempt to join an existing cluster 2024-02-27T06:15:42.304 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: using public_addr v2:172.21.15.150:0/0 -> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: starting mon.smithi150 rank -1 at public addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] at bind addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi150 fsid 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(???) e0 preinit fsid 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).mds e1 new map 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).mds e1 print_map 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: e1 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: legacy client fscid: -1 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: No filesystems configured 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-27T06:15:42.305 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: Deploying daemon crash.smithi131 on smithi131 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/783371298' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/783371298' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: osdmap e4: 0 total, 0 up, 0 in 2024-02-27T06:15:42.306 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: Deploying daemon grafana.smithi131 on smithi131 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi131:172.21.15.131=smithi131;smithi150:172.21.15.150=smithi150", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: Saving service mon spec with placement smithi131:172.21.15.131=smithi131;smithi150:172.21.15.150=smithi150;count:2 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/2317385540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/760605680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:42.307 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/2139969507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: Deploying daemon node-exporter.smithi131 on smithi131 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/3278462033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14160 172.21.15.131:0/147546065' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mgrmap e14: smithi131.frtgqo(active, since 46s) 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/565879410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/17183213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: Active manager daemon smithi131.frtgqo restarted 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: Activating manager daemon smithi131.frtgqo 2024-02-27T06:15:42.308 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: osdmap e5: 0 total, 0 up, 0 in 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mgrmap e15: smithi131.frtgqo(active, starting, since 0.0531725s) 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi131.frtgqo", "id": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: Manager daemon smithi131.frtgqo is now available 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: [27/Feb/2024:06:15:26] ENGINE Bus STARTING 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: [27/Feb/2024:06:15:26] ENGINE Serving on https://172.21.15.131:7150 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: [27/Feb/2024:06:15:26] ENGINE Bus STARTED 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mgrmap e16: smithi131.frtgqo(active, since 1.06148s) 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/2038090813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mgrmap e17: smithi131.frtgqo(active, since 2s) 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: Updating smithi131:/etc/ceph/ceph.conf 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd/host:smithi150", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.310 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: Deploying daemon crash.smithi150 on smithi150 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/2192797040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vvgbyr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vvgbyr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: Deploying daemon mgr.smithi150.vvgbyr on smithi150 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/837566533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:15:42.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:15:42.312 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: Deploying daemon mon.smithi150 on smithi150 2024-02-27T06:15:42.312 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:42.312 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-02-27T06:15:42.312 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 ceph-mon[26806]: mon.smithi150@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-27T06:15:42.312 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 bash[26763]: 976f96599f98d82552bd913ce1b6135c7ce3b0a452a56e634e3dad6fa75559f1 2024-02-27T06:15:42.312 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:42 smithi150 systemd[1]: Started Ceph mon.smithi150 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:15:47.379 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:15:47.379 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: mon.smithi131 calling monitor election 2024-02-27T06:15:47.379 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:47.379 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:47.379 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:47.379 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: mon.smithi150 calling monitor election 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/918699462' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/crt"}]: dispatch 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: mon.smithi131 is new leader, mons smithi131,smithi150 in quorum (ranks 0,1) 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: monmap e2: 2 mons at {smithi131=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],smithi150=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: fsmap 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: osdmap e5: 0 total, 0 up, 0 in 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: mgrmap e17: smithi131.frtgqo(active, since 21s) 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: Standby manager daemon smithi150.vvgbyr started 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/918699462' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: overall HEALTH_OK 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/918699462' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/key"}]: dispatch 2024-02-27T06:15:47.380 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/918699462' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:15:47.381 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:47 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:47.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:15:47.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: mon.smithi131 calling monitor election 2024-02-27T06:15:47.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:47.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:47.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:47.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: mon.smithi150 calling monitor election 2024-02-27T06:15:47.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/918699462' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/crt"}]: dispatch 2024-02-27T06:15:47.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:47.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:47.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:47.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:47.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: mon.smithi131 is new leader, mons smithi131,smithi150 in quorum (ranks 0,1) 2024-02-27T06:15:47.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: monmap e2: 2 mons at {smithi131=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],smithi150=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} 2024-02-27T06:15:47.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: fsmap 2024-02-27T06:15:47.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: osdmap e5: 0 total, 0 up, 0 in 2024-02-27T06:15:47.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: mgrmap e17: smithi131.frtgqo(active, since 21s) 2024-02-27T06:15:47.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: Standby manager daemon smithi150.vvgbyr started 2024-02-27T06:15:47.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/918699462' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:15:47.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: overall HEALTH_OK 2024-02-27T06:15:47.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/918699462' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/key"}]: dispatch 2024-02-27T06:15:47.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/918699462' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:15:47.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:47 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:48.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:48 smithi131 ceph-mon[24834]: Deploying daemon node-exporter.smithi150 on smithi150 2024-02-27T06:15:48.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:48 smithi131 ceph-mon[24834]: mgrmap e18: smithi131.frtgqo(active, since 21s), standbys: smithi150.vvgbyr 2024-02-27T06:15:48.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:48 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi150.vvgbyr", "id": "smithi150.vvgbyr"}]: dispatch 2024-02-27T06:15:48.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:48 smithi131 ceph-mon[24834]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:48.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:48 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:48.600 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:48 smithi150 ceph-mon[26806]: Deploying daemon node-exporter.smithi150 on smithi150 2024-02-27T06:15:48.600 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:48 smithi150 ceph-mon[26806]: mgrmap e18: smithi131.frtgqo(active, since 21s), standbys: smithi150.vvgbyr 2024-02-27T06:15:48.600 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:48 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi150.vvgbyr", "id": "smithi150.vvgbyr"}]: dispatch 2024-02-27T06:15:48.600 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:48 smithi150 ceph-mon[26806]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:48.600 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:48 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:15:48.776 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-02-27T06:15:48.776 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":2,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","modified":"2024-02-27T06:15:42.027262Z","created":"2024-02-27T06:13:38.291256Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi150","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-27T06:15:48.777 INFO:teuthology.orchestra.run.smithi150.stderr:dumped monmap epoch 2 2024-02-27T06:15:49.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:49 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/1339521112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:49.600 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:49 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/1339521112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-27T06:15:50.411 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:50 smithi150 ceph-mon[26806]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:50.435 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-27T06:15:50.436 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph config generate-minimal-conf 2024-02-27T06:15:50.467 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:50 smithi131 ceph-mon[24834]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:51.979 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:51 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:51.979 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:51 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:51.980 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:51 smithi131 ceph-mon[24834]: Deploying daemon prometheus.smithi131 on smithi131 2024-02-27T06:15:52.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:51 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:52.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:51 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:52.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:51 smithi150 ceph-mon[26806]: Deploying daemon prometheus.smithi131 on smithi131 2024-02-27T06:15:52.385 INFO:teuthology.orchestra.run.smithi131.stdout:# minimal ceph.conf for 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:15:52.385 INFO:teuthology.orchestra.run.smithi131.stdout:[global] 2024-02-27T06:15:52.385 INFO:teuthology.orchestra.run.smithi131.stdout: fsid = 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:15:52.385 INFO:teuthology.orchestra.run.smithi131.stdout: mon_host = [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-02-27T06:15:53.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:52 smithi131 ceph-mon[24834]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:53.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:52 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/616072330' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:15:53.150 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:52 smithi150 ceph-mon[26806]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:53.150 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:52 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/616072330' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:15:53.394 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-27T06:15:53.394 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-02-27T06:15:53.395 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.conf 2024-02-27T06:15:53.464 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-02-27T06:15:53.464 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:15:53.531 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-27T06:15:53.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.conf 2024-02-27T06:15:53.562 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-27T06:15:53.562 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:15:53.630 INFO:tasks.cephadm:Deploying OSDs... 2024-02-27T06:15:53.630 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-02-27T06:15:53.630 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2024-02-27T06:15:53.645 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-27T06:15:53.645 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_1 2024-02-27T06:15:53.699 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-27T06:15:53.700 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T06:15:53.700 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-27T06:15:53.700 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T06:15:53.700 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T06:15:53.700 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-02-27 06:15:16.528503397 +0000 2024-02-27T06:15:53.700 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-02-27 06:12:28.925824691 +0000 2024-02-27T06:15:53.700 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-02-27 06:12:28.925824691 +0000 2024-02-27T06:15:53.700 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-02-27 06:12:28.925824691 +0000 2024-02-27T06:15:53.700 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-27T06:15:53.766 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-02-27T06:15:53.766 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-02-27T06:15:53.766 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000253932 s, 2.0 MB/s 2024-02-27T06:15:53.768 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-27T06:15:53.823 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_2 2024-02-27T06:15:53.878 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-27T06:15:53.878 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T06:15:53.879 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-27T06:15:53.879 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T06:15:53.879 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T06:15:53.879 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-02-27 06:15:16.528503397 +0000 2024-02-27T06:15:53.879 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-02-27 06:12:29.279816949 +0000 2024-02-27T06:15:53.879 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-02-27 06:12:29.279816949 +0000 2024-02-27T06:15:53.879 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-02-27 06:12:29.279816949 +0000 2024-02-27T06:15:53.880 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-27T06:15:53.944 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-02-27T06:15:53.945 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-02-27T06:15:53.945 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.00017532 s, 2.9 MB/s 2024-02-27T06:15:53.946 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-27T06:15:54.002 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_3 2024-02-27T06:15:54.056 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-27T06:15:54.057 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T06:15:54.057 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-27T06:15:54.057 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T06:15:54.057 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T06:15:54.057 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-02-27 06:15:16.529503376 +0000 2024-02-27T06:15:54.057 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-02-27 06:12:29.613809644 +0000 2024-02-27T06:15:54.057 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-02-27 06:12:29.613809644 +0000 2024-02-27T06:15:54.057 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-02-27 06:12:29.613809644 +0000 2024-02-27T06:15:54.057 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-27T06:15:54.123 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-02-27T06:15:54.123 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-02-27T06:15:54.123 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000178122 s, 2.9 MB/s 2024-02-27T06:15:54.125 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-27T06:15:54.180 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_4 2024-02-27T06:15:54.234 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-27T06:15:54.234 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T06:15:54.235 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-27T06:15:54.235 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T06:15:54.235 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T06:15:54.235 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-02-27 06:15:16.529503376 +0000 2024-02-27T06:15:54.235 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-02-27 06:12:29.941802471 +0000 2024-02-27T06:15:54.235 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-02-27 06:12:29.941802471 +0000 2024-02-27T06:15:54.235 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-02-27 06:12:29.941802471 +0000 2024-02-27T06:15:54.235 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-27T06:15:54.300 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-02-27T06:15:54.300 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-02-27T06:15:54.300 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000172489 s, 3.0 MB/s 2024-02-27T06:15:54.301 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-27T06:15:54.356 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-02-27T06:15:54.356 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-02-27T06:15:54.375 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-27T06:15:54.375 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-02-27T06:15:54.432 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-27T06:15:54.433 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T06:15:54.433 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-27T06:15:54.433 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T06:15:54.433 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T06:15:54.433 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-27 06:15:47.670254644 +0000 2024-02-27T06:15:54.433 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-27 06:12:04.746296040 +0000 2024-02-27T06:15:54.433 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-27 06:12:04.746296040 +0000 2024-02-27T06:15:54.433 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-27 06:12:04.746296040 +0000 2024-02-27T06:15:54.434 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-27T06:15:54.501 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-27T06:15:54.501 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-27T06:15:54.501 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000231665 s, 2.2 MB/s 2024-02-27T06:15:54.503 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-27T06:15:54.560 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-02-27T06:15:54.617 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-27T06:15:54.617 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T06:15:54.617 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-27T06:15:54.617 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T06:15:54.617 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T06:15:54.617 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-27 06:15:47.671254624 +0000 2024-02-27T06:15:54.617 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-27 06:12:05.092289168 +0000 2024-02-27T06:15:54.617 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-27 06:12:05.092289168 +0000 2024-02-27T06:15:54.617 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-27 06:12:05.092289168 +0000 2024-02-27T06:15:54.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-27T06:15:54.684 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-27T06:15:54.684 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-27T06:15:54.684 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000226673 s, 2.3 MB/s 2024-02-27T06:15:54.688 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-27T06:15:54.745 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-02-27T06:15:54.800 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-27T06:15:54.800 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T06:15:54.801 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-27T06:15:54.801 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T06:15:54.801 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T06:15:54.801 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-27 06:15:47.671254624 +0000 2024-02-27T06:15:54.801 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-27 06:12:05.442282216 +0000 2024-02-27T06:15:54.801 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-27 06:12:05.442282216 +0000 2024-02-27T06:15:54.801 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-27 06:12:05.442282216 +0000 2024-02-27T06:15:54.801 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-27T06:15:54.867 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-27T06:15:54.867 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-27T06:15:54.868 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000226053 s, 2.3 MB/s 2024-02-27T06:15:54.869 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-27T06:15:54.925 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-02-27T06:15:54.980 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-27T06:15:54.980 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-27T06:15:54.980 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-27T06:15:54.981 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-27T06:15:54.981 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-02-27T06:15:54.981 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-02-27 06:15:47.671254624 +0000 2024-02-27T06:15:54.981 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-02-27 06:12:05.772275662 +0000 2024-02-27T06:15:54.981 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-02-27 06:12:05.772275662 +0000 2024-02-27T06:15:54.981 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-02-27 06:12:05.772275662 +0000 2024-02-27T06:15:54.981 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-27T06:15:55.047 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-02-27T06:15:55.047 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-02-27T06:15:55.047 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000225784 s, 2.3 MB/s 2024-02-27T06:15:55.048 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-27T06:15:55.103 INFO:tasks.cephadm:Deploying osd.0 on smithi131 with /dev/vg_nvme/lv_4... 2024-02-27T06:15:55.103 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-27T06:15:55.128 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:54 smithi131 ceph-mon[24834]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:55.350 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:54 smithi150 ceph-mon[26806]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:57.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:56 smithi131 ceph-mon[24834]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:57.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:56 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:57.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:56 smithi150 ceph-mon[26806]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:57.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:56 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:15:59.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:15:58 smithi131 ceph-mon[24834]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:15:59.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:15:58 smithi150 ceph-mon[26806]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:00.132 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-27T06:16:00.132 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-27T06:16:00.132 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-02-27T06:16:00.132 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-02-27T06:16:00.132 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011332 s, 925 MB/s 2024-02-27T06:16:00.132 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-02-27T06:16:01.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:00 smithi131 ceph-mon[24834]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:01.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:00 smithi150 ceph-mon[26806]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:02.062 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi131:vg_nvme/lv_4 2024-02-27T06:16:03.069 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:02 smithi131 ceph-mon[24834]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:03.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:02 smithi150 ceph-mon[26806]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:04.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:04 smithi131 ceph-mon[24834]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:05.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:04 smithi150 ceph-mon[26806]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:06.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:05 smithi131 ceph-mon[24834]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:16:06.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:05 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:16:06.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:05 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:16:06.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:05 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:06.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:05 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:06.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:05 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:16:06.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:05 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:06.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:05 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:16:06.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:05 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:06.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:05 smithi150 ceph-mon[26806]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:16:06.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:05 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:16:06.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:05 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:16:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:05 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:05 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:05 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:16:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:05 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:05 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:16:06.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:05 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:07.049 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:06 smithi150 ceph-mon[26806]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:07.063 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:06 smithi131 ceph-mon[24834]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:08.345 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:08.345 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:08 smithi131 ceph-mon[24834]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-27T06:16:08.345 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:08 smithi131 ceph-mon[24834]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:16:08.345 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:08.345 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:08 smithi131 ceph-mon[24834]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:08.350 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:08.350 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:08 smithi150 ceph-mon[26806]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-27T06:16:08.350 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:08 smithi150 ceph-mon[26806]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:16:08.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:08.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:08 smithi150 ceph-mon[26806]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:09.312 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:09 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/223554524' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff509ee5-7059-4672-b377-c3eb04d10814"}]: dispatch 2024-02-27T06:16:09.312 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:09 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/223554524' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff509ee5-7059-4672-b377-c3eb04d10814"}]': finished 2024-02-27T06:16:09.312 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:09 smithi131 ceph-mon[24834]: osdmap e6: 1 total, 0 up, 1 in 2024-02-27T06:16:09.312 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:09 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:16:09.350 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:09 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/223554524' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff509ee5-7059-4672-b377-c3eb04d10814"}]: dispatch 2024-02-27T06:16:09.350 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:09 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/223554524' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff509ee5-7059-4672-b377-c3eb04d10814"}]': finished 2024-02-27T06:16:09.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:09 smithi150 ceph-mon[26806]: osdmap e6: 1 total, 0 up, 1 in 2024-02-27T06:16:09.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:09 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:16:10.345 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:10 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/4203553134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:16:10.345 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:10 smithi131 ceph-mon[24834]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:10.350 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:10 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/4203553134' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:16:10.350 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:10 smithi150 ceph-mon[26806]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:13.006 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:12 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:13.006 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:12 smithi131 ceph-mon[24834]: Updating smithi131:/etc/ceph/ceph.conf 2024-02-27T06:16:13.006 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:12 smithi131 ceph-mon[24834]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:13.006 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:12 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:13.006 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:12 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:13.006 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:12 smithi131 ceph-mon[24834]: Reconfiguring mon.smithi131 (unknown last config time)... 2024-02-27T06:16:13.006 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:12 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:16:13.006 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:12 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T06:16:13.006 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:12 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:13.007 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:12 smithi131 ceph-mon[24834]: Reconfiguring daemon mon.smithi131 on smithi131 2024-02-27T06:16:13.054 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:12 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:13.054 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:12 smithi150 ceph-mon[26806]: Updating smithi131:/etc/ceph/ceph.conf 2024-02-27T06:16:13.054 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:12 smithi150 ceph-mon[26806]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:13.054 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:12 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:13.054 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:12 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:13.054 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:12 smithi150 ceph-mon[26806]: Reconfiguring mon.smithi131 (unknown last config time)... 2024-02-27T06:16:13.054 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:12 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:16:13.054 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:12 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T06:16:13.054 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:12 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:13.055 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:12 smithi150 ceph-mon[26806]: Reconfiguring daemon mon.smithi131 on smithi131 2024-02-27T06:16:14.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:14 smithi131 ceph-mon[24834]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:14.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:14 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:14.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:14 smithi131 ceph-mon[24834]: Reconfiguring mgr.smithi131.frtgqo (unknown last config time)... 2024-02-27T06:16:14.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:14 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi131.frtgqo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:16:14.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:14 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:16:14.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:14 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:14.961 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:14 smithi131 ceph-mon[24834]: Reconfiguring daemon mgr.smithi131.frtgqo on smithi131 2024-02-27T06:16:15.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:14 smithi150 ceph-mon[26806]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:15.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:14 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:15.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:14 smithi150 ceph-mon[26806]: Reconfiguring mgr.smithi131.frtgqo (unknown last config time)... 2024-02-27T06:16:15.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:14 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi131.frtgqo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:16:15.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:14 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:16:15.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:14 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:15.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:14 smithi150 ceph-mon[26806]: Reconfiguring daemon mgr.smithi131.frtgqo on smithi131 2024-02-27T06:16:17.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:16 smithi131 ceph-mon[24834]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:17.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:16 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:17.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:16 smithi131 ceph-mon[24834]: Reconfiguring alertmanager.smithi131 (dependencies changed)... 2024-02-27T06:16:17.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:16 smithi131 ceph-mon[24834]: Reconfiguring daemon alertmanager.smithi131 on smithi131 2024-02-27T06:16:17.350 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:16 smithi150 ceph-mon[26806]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:17.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:16 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:17.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:16 smithi150 ceph-mon[26806]: Reconfiguring alertmanager.smithi131 (dependencies changed)... 2024-02-27T06:16:17.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:16 smithi150 ceph-mon[26806]: Reconfiguring daemon alertmanager.smithi131 on smithi131 2024-02-27T06:16:18.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:17 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T06:16:18.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:17 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:18.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:17 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T06:16:18.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:17 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:19.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:18 smithi131 ceph-mon[24834]: Deploying daemon osd.0 on smithi131 2024-02-27T06:16:19.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:18 smithi131 ceph-mon[24834]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:19.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:18 smithi150 ceph-mon[26806]: Deploying daemon osd.0 on smithi131 2024-02-27T06:16:19.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:18 smithi150 ceph-mon[26806]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:21.271 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:20 smithi131 ceph-mon[24834]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:21.350 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:20 smithi150 ceph-mon[26806]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:22.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:22.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:22 smithi131 ceph-mon[24834]: Reconfiguring crash.smithi131 (monmap changed)... 2024-02-27T06:16:22.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:16:22.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:22.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:22 smithi131 ceph-mon[24834]: Reconfiguring daemon crash.smithi131 on smithi131 2024-02-27T06:16:22.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:22 smithi131 ceph-mon[24834]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:22.750 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:22.751 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:22 smithi150 ceph-mon[26806]: Reconfiguring crash.smithi131 (monmap changed)... 2024-02-27T06:16:22.751 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:16:22.751 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:22.751 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:22 smithi150 ceph-mon[26806]: Reconfiguring daemon crash.smithi131 on smithi131 2024-02-27T06:16:22.751 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:22 smithi150 ceph-mon[26806]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:25.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:24 smithi131 ceph-mon[24834]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:25.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:24 smithi150 ceph-mon[26806]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:25.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:16:26.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:16:26.933 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:26 smithi131 ceph-mon[24834]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:26.933 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:16:26.933 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:27.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:26 smithi150 ceph-mon[26806]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:27.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:16:27.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:28.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:28 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:28.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:28 smithi131 ceph-mon[24834]: Reconfiguring grafana.smithi131 (dependencies changed)... 2024-02-27T06:16:28.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:28 smithi131 ceph-mon[24834]: Reconfiguring daemon grafana.smithi131 on smithi131 2024-02-27T06:16:28.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:28 smithi131 ceph-mon[24834]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:29.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:28 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:29.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:28 smithi150 ceph-mon[26806]: Reconfiguring grafana.smithi131 (dependencies changed)... 2024-02-27T06:16:29.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:28 smithi150 ceph-mon[26806]: Reconfiguring daemon grafana.smithi131 on smithi131 2024-02-27T06:16:29.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:28 smithi150 ceph-mon[26806]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:29.800 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:29 smithi131 ceph-mon[24834]: from='osd.0 [v2:172.21.15.131:6802/3100254707,v1:172.21.15.131:6803/3100254707]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T06:16:30.100 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:29 smithi150 ceph-mon[26806]: from='osd.0 [v2:172.21.15.131:6802/3100254707,v1:172.21.15.131:6803/3100254707]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T06:16:30.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:30 smithi131 ceph-mon[24834]: from='osd.0 [v2:172.21.15.131:6802/3100254707,v1:172.21.15.131:6803/3100254707]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-27T06:16:30.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:30 smithi131 ceph-mon[24834]: osdmap e7: 1 total, 0 up, 1 in 2024-02-27T06:16:30.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:30 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:16:30.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:30 smithi131 ceph-mon[24834]: from='osd.0 [v2:172.21.15.131:6802/3100254707,v1:172.21.15.131:6803/3100254707]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:16:30.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:30 smithi131 ceph-mon[24834]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:31.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:30 smithi150 ceph-mon[26806]: from='osd.0 [v2:172.21.15.131:6802/3100254707,v1:172.21.15.131:6803/3100254707]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-27T06:16:31.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:30 smithi150 ceph-mon[26806]: osdmap e7: 1 total, 0 up, 1 in 2024-02-27T06:16:31.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:30 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:16:31.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:30 smithi150 ceph-mon[26806]: from='osd.0 [v2:172.21.15.131:6802/3100254707,v1:172.21.15.131:6803/3100254707]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:16:31.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:30 smithi150 ceph-mon[26806]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:32.017 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:31 smithi131 ceph-mon[24834]: from='osd.0 [v2:172.21.15.131:6802/3100254707,v1:172.21.15.131:6803/3100254707]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-02-27T06:16:32.017 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:31 smithi131 ceph-mon[24834]: osdmap e8: 1 total, 0 up, 1 in 2024-02-27T06:16:32.017 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:31 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:16:32.017 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:31 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:16:32.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:31 smithi150 ceph-mon[26806]: from='osd.0 [v2:172.21.15.131:6802/3100254707,v1:172.21.15.131:6803/3100254707]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-02-27T06:16:32.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:31 smithi150 ceph-mon[26806]: osdmap e8: 1 total, 0 up, 1 in 2024-02-27T06:16:32.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:31 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:16:32.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:31 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:16:32.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:32 smithi131 ceph-mon[24834]: purged_snaps scrub starts 2024-02-27T06:16:32.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:32 smithi131 ceph-mon[24834]: purged_snaps scrub ok 2024-02-27T06:16:32.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:32 smithi131 ceph-mon[24834]: osd.0 [v2:172.21.15.131:6802/3100254707,v1:172.21.15.131:6803/3100254707] boot 2024-02-27T06:16:32.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:32 smithi131 ceph-mon[24834]: osdmap e9: 1 total, 1 up, 1 in 2024-02-27T06:16:32.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:32 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:16:32.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:32 smithi131 ceph-mon[24834]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:32.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:32 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:32.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:32 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:16:32.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:32 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:33.030 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:32 smithi150 ceph-mon[26806]: purged_snaps scrub starts 2024-02-27T06:16:33.031 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:32 smithi150 ceph-mon[26806]: purged_snaps scrub ok 2024-02-27T06:16:33.031 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:32 smithi150 ceph-mon[26806]: osd.0 [v2:172.21.15.131:6802/3100254707,v1:172.21.15.131:6803/3100254707] boot 2024-02-27T06:16:33.031 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:32 smithi150 ceph-mon[26806]: osdmap e9: 1 total, 1 up, 1 in 2024-02-27T06:16:33.031 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:32 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:16:33.031 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:32 smithi150 ceph-mon[26806]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-27T06:16:33.031 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:32 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:33.031 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:32 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:16:33.031 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:32 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:33.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:33 smithi131 ceph-mon[24834]: Reconfiguring crash.smithi150 (monmap changed)... 2024-02-27T06:16:33.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:33 smithi131 ceph-mon[24834]: Reconfiguring daemon crash.smithi150 on smithi150 2024-02-27T06:16:33.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:33 smithi131 ceph-mon[24834]: osdmap e10: 1 total, 1 up, 1 in 2024-02-27T06:16:33.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:33 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:33.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:33 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vvgbyr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:16:33.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:33 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:16:33.792 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:33 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:33.822 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 0 on host 'smithi131' 2024-02-27T06:16:33.851 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:33 smithi150 ceph-mon[26806]: Reconfiguring crash.smithi150 (monmap changed)... 2024-02-27T06:16:33.851 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:33 smithi150 ceph-mon[26806]: Reconfiguring daemon crash.smithi150 on smithi150 2024-02-27T06:16:33.851 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:33 smithi150 ceph-mon[26806]: osdmap e10: 1 total, 1 up, 1 in 2024-02-27T06:16:33.851 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:33 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:33.851 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:33 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vvgbyr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:16:33.851 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:33 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:16:33.851 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:33 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:34.436 DEBUG:teuthology.orchestra.run.smithi131:osd.0> sudo journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.0.service 2024-02-27T06:16:34.438 INFO:tasks.cephadm:Deploying osd.1 on smithi131 with /dev/vg_nvme/lv_3... 2024-02-27T06:16:34.439 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-27T06:16:35.082 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:34 smithi131 ceph-mon[24834]: Reconfiguring mgr.smithi150.vvgbyr (monmap changed)... 2024-02-27T06:16:35.082 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:34 smithi131 ceph-mon[24834]: Reconfiguring daemon mgr.smithi150.vvgbyr on smithi150 2024-02-27T06:16:35.082 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:34 smithi131 ceph-mon[24834]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:35.083 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:35.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:34 smithi150 ceph-mon[26806]: Reconfiguring mgr.smithi150.vvgbyr (monmap changed)... 2024-02-27T06:16:35.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:34 smithi150 ceph-mon[26806]: Reconfiguring daemon mgr.smithi150.vvgbyr on smithi150 2024-02-27T06:16:35.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:34 smithi150 ceph-mon[26806]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:35.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:36.091 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:35 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:36.091 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:35 smithi131 ceph-mon[24834]: Reconfiguring mon.smithi150 (monmap changed)... 2024-02-27T06:16:36.091 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:35 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:16:36.091 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:35 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T06:16:36.091 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:35 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:36.091 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:35 smithi131 ceph-mon[24834]: Reconfiguring daemon mon.smithi150 on smithi150 2024-02-27T06:16:36.212 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:35 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:36.212 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:35 smithi150 ceph-mon[26806]: Reconfiguring mon.smithi150 (monmap changed)... 2024-02-27T06:16:36.212 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:35 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:16:36.212 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:35 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T06:16:36.212 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:35 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:36.212 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:35 smithi150 ceph-mon[26806]: Reconfiguring daemon mon.smithi150 on smithi150 2024-02-27T06:16:36.478 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-27T06:16:36.478 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-27T06:16:36.478 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-02-27T06:16:36.478 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-02-27T06:16:36.478 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114112 s, 919 MB/s 2024-02-27T06:16:36.478 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-02-27T06:16:37.096 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi131:vg_nvme/lv_3 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.131:9093"}]: dispatch 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.131:3000"}]: dispatch 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.131:9095"}]: dispatch 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:16:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:37.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:16:37.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:37.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:37.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T06:16:37.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.131:9093"}]: dispatch 2024-02-27T06:16:37.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:37.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T06:16:37.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.131:3000"}]: dispatch 2024-02-27T06:16:37.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:37.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:16:37.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.131:9095"}]: dispatch 2024-02-27T06:16:37.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:37.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:16:37.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:37.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:16:38.236 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:37 smithi131 ceph-mon[24834]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T06:16:38.237 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:37 smithi131 ceph-mon[24834]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.131:9093"}]: dispatch 2024-02-27T06:16:38.237 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:37 smithi131 ceph-mon[24834]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T06:16:38.237 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:37 smithi131 ceph-mon[24834]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.131:3000"}]: dispatch 2024-02-27T06:16:38.237 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:37 smithi131 ceph-mon[24834]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:16:38.237 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:37 smithi131 ceph-mon[24834]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.131:9095"}]: dispatch 2024-02-27T06:16:38.328 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:37 smithi150 ceph-mon[26806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T06:16:38.328 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:37 smithi150 ceph-mon[26806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.131:9093"}]: dispatch 2024-02-27T06:16:38.328 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:37 smithi150 ceph-mon[26806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T06:16:38.329 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:37 smithi150 ceph-mon[26806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.131:3000"}]: dispatch 2024-02-27T06:16:38.329 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:37 smithi150 ceph-mon[26806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:16:38.329 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:37 smithi150 ceph-mon[26806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.131:9095"}]: dispatch 2024-02-27T06:16:39.212 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:38 smithi131 ceph-mon[24834]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:39.212 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:38 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:39.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:38 smithi150 ceph-mon[26806]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:39.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:38 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:40.222 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:39 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:16:40.222 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:39 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:16:40.222 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:39 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:40.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:39 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:16:40.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:39 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:16:40.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:39 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:41.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:40 smithi131 ceph-mon[24834]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:16:41.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:40 smithi131 ceph-mon[24834]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:41.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:40 smithi150 ceph-mon[26806]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:16:41.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:40 smithi150 ceph-mon[26806]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:43.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:42 smithi131 ceph-mon[24834]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:43.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:42 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1630098848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3aa96ce-7431-4a6f-adca-92a510c205f1"}]: dispatch 2024-02-27T06:16:43.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:42 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1630098848' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3aa96ce-7431-4a6f-adca-92a510c205f1"}]': finished 2024-02-27T06:16:43.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:42 smithi131 ceph-mon[24834]: osdmap e11: 2 total, 1 up, 2 in 2024-02-27T06:16:43.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:42 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:16:43.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:42 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/4099518576' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:16:43.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:42 smithi150 ceph-mon[26806]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:43.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1630098848' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3aa96ce-7431-4a6f-adca-92a510c205f1"}]: dispatch 2024-02-27T06:16:43.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1630098848' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3aa96ce-7431-4a6f-adca-92a510c205f1"}]': finished 2024-02-27T06:16:43.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:42 smithi150 ceph-mon[26806]: osdmap e11: 2 total, 1 up, 2 in 2024-02-27T06:16:43.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:42 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:16:43.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/4099518576' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:16:45.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:44 smithi131 ceph-mon[24834]: pgmap v38: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:45.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:44 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:45.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:44 smithi150 ceph-mon[26806]: pgmap v38: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:45.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:44 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:46.963 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:46 smithi131 ceph-mon[24834]: pgmap v39: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:47.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:46 smithi150 ceph-mon[26806]: pgmap v39: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:49.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:48 smithi131 ceph-mon[24834]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:49.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:48 smithi131 ceph-mon[24834]: Detected new or changed devices on smithi131 2024-02-27T06:16:49.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:48 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:49.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:48 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:16:49.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:48 smithi131 ceph-mon[24834]: Adjusting osd_memory_target on smithi131 to 12864M 2024-02-27T06:16:49.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:48 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:49.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:48 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:49.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:48 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:16:49.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:48 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:49.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:48 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:16:49.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:48 smithi150 ceph-mon[26806]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:49.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:48 smithi150 ceph-mon[26806]: Detected new or changed devices on smithi131 2024-02-27T06:16:49.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:48 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:49.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:48 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:16:49.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:48 smithi150 ceph-mon[26806]: Adjusting osd_memory_target on smithi131 to 12864M 2024-02-27T06:16:49.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:48 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:49.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:48 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:49.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:48 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:16:49.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:48 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:49.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:48 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:16:50.845 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:50 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:50.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:50 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T06:16:50.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:50 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:50.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:50 smithi131 ceph-mon[24834]: Deploying daemon osd.1 on smithi131 2024-02-27T06:16:50.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:50 smithi131 ceph-mon[24834]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:50.851 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:50 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:50.851 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:50 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T06:16:50.851 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:50 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:50.851 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:50 smithi150 ceph-mon[26806]: Deploying daemon osd.1 on smithi131 2024-02-27T06:16:50.851 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:50 smithi150 ceph-mon[26806]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:53.085 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:52 smithi131 ceph-mon[24834]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:53.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:52 smithi150 ceph-mon[26806]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:55.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:54 smithi131 ceph-mon[24834]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:55.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:54 smithi150 ceph-mon[26806]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:56.236 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:55 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:56.236 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:55 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:56.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:55 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:56.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:55 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:57.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:56 smithi131 ceph-mon[24834]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:57.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:56 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:57.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:56 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:16:57.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:56 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:57.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:56 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:16:57.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:56 smithi150 ceph-mon[26806]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:57.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:56 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:57.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:56 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:16:57.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:56 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:16:57.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:56 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:16:59.048 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:58 smithi131 ceph-mon[24834]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:59.048 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:58 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:16:59.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:58 smithi150 ceph-mon[26806]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:16:59.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:58 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:00.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:16:59 smithi131 ceph-mon[24834]: from='osd.1 [v2:172.21.15.131:6810/1508562642,v1:172.21.15.131:6811/1508562642]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T06:17:00.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:16:59 smithi150 ceph-mon[26806]: from='osd.1 [v2:172.21.15.131:6810/1508562642,v1:172.21.15.131:6811/1508562642]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T06:17:00.683 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 1 on host 'smithi131' 2024-02-27T06:17:01.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:00 smithi131 ceph-mon[24834]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:17:01.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:00 smithi131 ceph-mon[24834]: from='osd.1 [v2:172.21.15.131:6810/1508562642,v1:172.21.15.131:6811/1508562642]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-27T06:17:01.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:00 smithi131 ceph-mon[24834]: osdmap e12: 2 total, 1 up, 2 in 2024-02-27T06:17:01.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:17:01.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:00 smithi131 ceph-mon[24834]: from='osd.1 [v2:172.21.15.131:6810/1508562642,v1:172.21.15.131:6811/1508562642]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:17:01.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:01.239 DEBUG:teuthology.orchestra.run.smithi131:osd.1> sudo journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.1.service 2024-02-27T06:17:01.241 INFO:tasks.cephadm:Deploying osd.2 on smithi131 with /dev/vg_nvme/lv_2... 2024-02-27T06:17:01.241 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-27T06:17:01.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:00 smithi150 ceph-mon[26806]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:17:01.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:00 smithi150 ceph-mon[26806]: from='osd.1 [v2:172.21.15.131:6810/1508562642,v1:172.21.15.131:6811/1508562642]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-27T06:17:01.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:00 smithi150 ceph-mon[26806]: osdmap e12: 2 total, 1 up, 2 in 2024-02-27T06:17:01.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:17:01.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:00 smithi150 ceph-mon[26806]: from='osd.1 [v2:172.21.15.131:6810/1508562642,v1:172.21.15.131:6811/1508562642]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:17:01.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:02.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:01 smithi131 ceph-mon[24834]: from='osd.1 [v2:172.21.15.131:6810/1508562642,v1:172.21.15.131:6811/1508562642]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-02-27T06:17:02.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:01 smithi131 ceph-mon[24834]: osdmap e13: 2 total, 1 up, 2 in 2024-02-27T06:17:02.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:01 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:17:02.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:01 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:17:02.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:01 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:17:02.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:01 smithi150 ceph-mon[26806]: from='osd.1 [v2:172.21.15.131:6810/1508562642,v1:172.21.15.131:6811/1508562642]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-02-27T06:17:02.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:01 smithi150 ceph-mon[26806]: osdmap e13: 2 total, 1 up, 2 in 2024-02-27T06:17:02.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:01 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:17:02.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:01 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:17:02.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:01 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:17:03.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:02 smithi150 ceph-mon[26806]: purged_snaps scrub starts 2024-02-27T06:17:03.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:02 smithi150 ceph-mon[26806]: purged_snaps scrub ok 2024-02-27T06:17:03.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:02 smithi150 ceph-mon[26806]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:17:03.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:02 smithi150 ceph-mon[26806]: osd.1 [v2:172.21.15.131:6810/1508562642,v1:172.21.15.131:6811/1508562642] boot 2024-02-27T06:17:03.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:02 smithi150 ceph-mon[26806]: osdmap e14: 2 total, 2 up, 2 in 2024-02-27T06:17:03.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:02 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:17:03.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:02 smithi131 ceph-mon[24834]: purged_snaps scrub starts 2024-02-27T06:17:03.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:02 smithi131 ceph-mon[24834]: purged_snaps scrub ok 2024-02-27T06:17:03.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:02 smithi131 ceph-mon[24834]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-02-27T06:17:03.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:02 smithi131 ceph-mon[24834]: osd.1 [v2:172.21.15.131:6810/1508562642,v1:172.21.15.131:6811/1508562642] boot 2024-02-27T06:17:03.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:02 smithi131 ceph-mon[24834]: osdmap e14: 2 total, 2 up, 2 in 2024-02-27T06:17:03.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:02 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:17:04.274 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:03 smithi131 ceph-mon[24834]: osdmap e15: 2 total, 2 up, 2 in 2024-02-27T06:17:04.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:03 smithi150 ceph-mon[26806]: osdmap e15: 2 total, 2 up, 2 in 2024-02-27T06:17:04.919 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-27T06:17:04.920 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-27T06:17:04.920 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-02-27T06:17:04.920 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-02-27T06:17:04.920 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128928 s, 813 MB/s 2024-02-27T06:17:04.920 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-02-27T06:17:05.252 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:04 smithi131 ceph-mon[24834]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:05.252 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:04 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:05.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:04 smithi150 ceph-mon[26806]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:05.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:04 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:05.532 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi131:vg_nvme/lv_2 2024-02-27T06:17:07.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:06 smithi131 ceph-mon[24834]: pgmap v53: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:07.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:06 smithi150 ceph-mon[26806]: pgmap v53: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:09.087 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:08 smithi131 ceph-mon[24834]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:09.087 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:17:09.087 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:17:09.087 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:09.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:08 smithi150 ceph-mon[26806]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:09.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:17:09.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:17:09.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:10.233 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:09 smithi131 ceph-mon[24834]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:17:10.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:09 smithi150 ceph-mon[26806]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:17:11.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:10 smithi131 ceph-mon[24834]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:11.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:11.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:17:11.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:17:11.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:11.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:11.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:17:11.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:11.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:17:11.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:10 smithi150 ceph-mon[26806]: pgmap v55: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:11.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:11.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:17:11.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:17:11.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:11.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:11.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:17:11.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:11.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:17:12.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:11 smithi131 ceph-mon[24834]: Adjusting osd_memory_target on smithi131 to 6432M 2024-02-27T06:17:12.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:11 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/3385568773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6fb48b20-5ecd-4d7c-8784-b0fea58df271"}]: dispatch 2024-02-27T06:17:12.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:11 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/3385568773' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fb48b20-5ecd-4d7c-8784-b0fea58df271"}]': finished 2024-02-27T06:17:12.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:11 smithi131 ceph-mon[24834]: osdmap e16: 3 total, 2 up, 3 in 2024-02-27T06:17:12.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:11 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:17:12.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:11 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2731974459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:17:12.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:11 smithi150 ceph-mon[26806]: Adjusting osd_memory_target on smithi131 to 6432M 2024-02-27T06:17:12.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:11 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/3385568773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6fb48b20-5ecd-4d7c-8784-b0fea58df271"}]: dispatch 2024-02-27T06:17:12.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:11 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/3385568773' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fb48b20-5ecd-4d7c-8784-b0fea58df271"}]': finished 2024-02-27T06:17:12.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:11 smithi150 ceph-mon[26806]: osdmap e16: 3 total, 2 up, 3 in 2024-02-27T06:17:12.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:11 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:17:12.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:11 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/2731974459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:17:13.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:12 smithi150 ceph-mon[26806]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:13.324 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:12 smithi131 ceph-mon[24834]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:15.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:14 smithi131 ceph-mon[24834]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:15.245 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:14 smithi150 ceph-mon[26806]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:16.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:16 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:16.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:16 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:16.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:16 smithi131 ceph-mon[24834]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:16.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:16 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:16.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:16 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:16.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:16 smithi150 ceph-mon[26806]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:18.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:17 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T06:17:18.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:17 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:18.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:17 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T06:17:18.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:17 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:19.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:18 smithi131 ceph-mon[24834]: Deploying daemon osd.2 on smithi131 2024-02-27T06:17:19.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:18 smithi131 ceph-mon[24834]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:19.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:18 smithi150 ceph-mon[26806]: Deploying daemon osd.2 on smithi131 2024-02-27T06:17:19.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:18 smithi150 ceph-mon[26806]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:21.001 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:20 smithi131 ceph-mon[24834]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:21.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:20 smithi150 ceph-mon[26806]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:22.971 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:22 smithi131 ceph-mon[24834]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:22.971 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:22.971 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:17:22.971 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:22.971 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:17:23.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:22 smithi150 ceph-mon[26806]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:23.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:23.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:17:23.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:23.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:17:25.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:24 smithi150 ceph-mon[26806]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:25.138 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:24 smithi131 ceph-mon[24834]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:26.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:25 smithi131 ceph-mon[24834]: from='osd.2 [v2:172.21.15.131:6818/567909672,v1:172.21.15.131:6819/567909672]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T06:17:26.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:17:26.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:17:26.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:25 smithi150 ceph-mon[26806]: from='osd.2 [v2:172.21.15.131:6818/567909672,v1:172.21.15.131:6819/567909672]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T06:17:26.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:17:26.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:17:26.550 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 2 on host 'smithi131' 2024-02-27T06:17:27.189 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:26 smithi131 ceph-mon[24834]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:27.189 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:26 smithi131 ceph-mon[24834]: from='osd.2 [v2:172.21.15.131:6818/567909672,v1:172.21.15.131:6819/567909672]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-27T06:17:27.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:26 smithi131 ceph-mon[24834]: osdmap e17: 3 total, 2 up, 3 in 2024-02-27T06:17:27.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:17:27.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:26 smithi131 ceph-mon[24834]: from='osd.2 [v2:172.21.15.131:6818/567909672,v1:172.21.15.131:6819/567909672]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:17:27.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:27.213 DEBUG:teuthology.orchestra.run.smithi131:osd.2> sudo journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.2.service 2024-02-27T06:17:27.215 INFO:tasks.cephadm:Deploying osd.3 on smithi131 with /dev/vg_nvme/lv_1... 2024-02-27T06:17:27.216 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-27T06:17:27.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:26 smithi150 ceph-mon[26806]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:27.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:26 smithi150 ceph-mon[26806]: from='osd.2 [v2:172.21.15.131:6818/567909672,v1:172.21.15.131:6819/567909672]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-27T06:17:27.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:26 smithi150 ceph-mon[26806]: osdmap e17: 3 total, 2 up, 3 in 2024-02-27T06:17:27.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:17:27.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:26 smithi150 ceph-mon[26806]: from='osd.2 [v2:172.21.15.131:6818/567909672,v1:172.21.15.131:6819/567909672]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:17:27.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:28.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:27 smithi131 ceph-mon[24834]: from='osd.2 [v2:172.21.15.131:6818/567909672,v1:172.21.15.131:6819/567909672]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-02-27T06:17:28.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:27 smithi131 ceph-mon[24834]: osdmap e18: 3 total, 2 up, 3 in 2024-02-27T06:17:28.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:27 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:17:28.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:27 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:17:28.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:27 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:17:28.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:27 smithi150 ceph-mon[26806]: from='osd.2 [v2:172.21.15.131:6818/567909672,v1:172.21.15.131:6819/567909672]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-02-27T06:17:28.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:27 smithi150 ceph-mon[26806]: osdmap e18: 3 total, 2 up, 3 in 2024-02-27T06:17:28.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:27 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:17:28.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:27 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:17:28.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:27 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:17:28.959 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:28 smithi131 ceph-mon[24834]: purged_snaps scrub starts 2024-02-27T06:17:28.959 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:28 smithi131 ceph-mon[24834]: purged_snaps scrub ok 2024-02-27T06:17:28.959 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:28 smithi131 ceph-mon[24834]: pgmap v67: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:28.959 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:28 smithi131 ceph-mon[24834]: osd.2 [v2:172.21.15.131:6818/567909672,v1:172.21.15.131:6819/567909672] boot 2024-02-27T06:17:28.959 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:28 smithi131 ceph-mon[24834]: osdmap e19: 3 total, 3 up, 3 in 2024-02-27T06:17:28.959 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:28 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:17:29.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:28 smithi150 ceph-mon[26806]: purged_snaps scrub starts 2024-02-27T06:17:29.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:28 smithi150 ceph-mon[26806]: purged_snaps scrub ok 2024-02-27T06:17:29.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:28 smithi150 ceph-mon[26806]: pgmap v67: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-02-27T06:17:29.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:28 smithi150 ceph-mon[26806]: osd.2 [v2:172.21.15.131:6818/567909672,v1:172.21.15.131:6819/567909672] boot 2024-02-27T06:17:29.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:28 smithi150 ceph-mon[26806]: osdmap e19: 3 total, 3 up, 3 in 2024-02-27T06:17:29.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:28 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:17:29.790 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-27T06:17:29.790 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-27T06:17:29.790 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-02-27T06:17:29.790 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-02-27T06:17:29.790 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122777 s, 854 MB/s 2024-02-27T06:17:29.790 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-02-27T06:17:30.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:30 smithi131 ceph-mon[24834]: osdmap e20: 3 total, 3 up, 3 in 2024-02-27T06:17:30.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:30 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-27T06:17:30.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:30 smithi150 ceph-mon[26806]: osdmap e20: 3 total, 3 up, 3 in 2024-02-27T06:17:30.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:30 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-27T06:17:30.407 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi131:vg_nvme/lv_1 2024-02-27T06:17:31.314 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:31 smithi131 ceph-mon[24834]: pgmap v70: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:31.314 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:31 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-27T06:17:31.314 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:31 smithi131 ceph-mon[24834]: osdmap e21: 3 total, 3 up, 3 in 2024-02-27T06:17:31.314 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:31 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-27T06:17:31.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:31 smithi150 ceph-mon[26806]: pgmap v70: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:31.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:31 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-27T06:17:31.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:31 smithi150 ceph-mon[26806]: osdmap e21: 3 total, 3 up, 3 in 2024-02-27T06:17:31.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:31 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-27T06:17:31.962 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:17:31 smithi131 sudo[44316]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-27T06:17:31.962 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:17:31 smithi131 sudo[44316]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T06:17:31.962 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:17:31 smithi131 sudo[44316]: pam_unix(sudo:session): session closed for user root 2024-02-27T06:17:31.962 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:17:31 smithi131 sudo[44319]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-27T06:17:31.962 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:17:31 smithi131 sudo[44319]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T06:17:32.231 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:17:31 smithi131 sudo[44319]: pam_unix(sudo:session): session closed for user root 2024-02-27T06:17:32.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:32 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-27T06:17:32.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:32 smithi131 ceph-mon[24834]: osdmap e22: 3 total, 3 up, 3 in 2024-02-27T06:17:32.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:32 smithi131 ceph-mon[24834]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:32.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:32 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:32.491 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:17:32 smithi131 sudo[44361]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-27T06:17:32.491 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:17:32 smithi131 sudo[44361]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T06:17:32.492 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:17:32 smithi131 sudo[44361]: pam_unix(sudo:session): session closed for user root 2024-02-27T06:17:32.492 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:17:32 smithi131 sudo[44370]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-27T06:17:32.601 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:32 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-27T06:17:32.602 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:32 smithi150 ceph-mon[26806]: osdmap e22: 3 total, 3 up, 3 in 2024-02-27T06:17:32.602 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:32 smithi150 ceph-mon[26806]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:32.602 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:32 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:32.743 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:17:32 smithi131 sudo[44370]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T06:17:32.744 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:17:32 smithi131 sudo[44370]: pam_unix(sudo:session): session closed for user root 2024-02-27T06:17:33.003 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:17:32 smithi131 sudo[44428]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-27T06:17:33.004 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:17:32 smithi131 sudo[44428]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T06:17:33.004 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:17:32 smithi131 sudo[44428]: pam_unix(sudo:session): session closed for user root 2024-02-27T06:17:33.262 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:17:33 smithi131 sudo[44470]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-27T06:17:33.262 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:17:33 smithi131 sudo[44470]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T06:17:33.262 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:17:33 smithi131 sudo[44470]: pam_unix(sudo:session): session closed for user root 2024-02-27T06:17:33.518 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:33 smithi131 sudo[44532]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-27T06:17:33.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:33 smithi131 sudo[44532]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T06:17:33.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:33 smithi131 sudo[44532]: pam_unix(sudo:session): session closed for user root 2024-02-27T06:17:33.768 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:33 smithi131 sudo[44539]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-27T06:17:33.768 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:33 smithi131 sudo[44539]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T06:17:33.769 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:33 smithi131 sudo[44539]: pam_unix(sudo:session): session closed for user root 2024-02-27T06:17:33.769 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:33 smithi131 ceph-mon[24834]: osdmap e23: 3 total, 3 up, 3 in 2024-02-27T06:17:33.769 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:33 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:33.782 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:33 smithi150 ceph-mon[26806]: osdmap e23: 3 total, 3 up, 3 in 2024-02-27T06:17:33.782 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:33 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:33.782 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:33 smithi150 sudo[29044]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-27T06:17:33.783 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:33 smithi150 sudo[29044]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T06:17:33.783 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:33 smithi150 sudo[29044]: pam_unix(sudo:session): session closed for user root 2024-02-27T06:17:34.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 sudo[29047]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-27T06:17:34.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 sudo[29047]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-27T06:17:34.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 sudo[29047]: pam_unix(sudo:session): session closed for user root 2024-02-27T06:17:34.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T06:17:34.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T06:17:34.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:17:34.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:17:34.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T06:17:34.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: mgrmap e19: smithi131.frtgqo(active, since 2m), standbys: smithi150.vvgbyr 2024-02-27T06:17:34.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:17:34.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:17:34.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:17:34.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:17:34.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:17:34.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:34.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:34.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T06:17:34.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:34.852 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T06:17:34.852 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T06:17:34.852 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:17:34.852 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:17:34.852 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-27T06:17:34.852 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: mgrmap e19: smithi131.frtgqo(active, since 2m), standbys: smithi150.vvgbyr 2024-02-27T06:17:34.852 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:17:34.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:17:34.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:17:34.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:17:34.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:17:34.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:34.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:34.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-27T06:17:34.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:37.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:36 smithi131 ceph-mon[24834]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:37.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:37.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:17:37.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:17:37.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:17:37.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:37.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:37.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:17:37.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:37.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:17:37.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:36 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:37.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:36 smithi150 ceph-mon[26806]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:37.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:37.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:17:37.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:17:37.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:17:37.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:37.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:37.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:17:37.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:37.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:17:37.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:36 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:38.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:37 smithi131 ceph-mon[24834]: Detected new or changed devices on smithi131 2024-02-27T06:17:38.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:37 smithi131 ceph-mon[24834]: Adjusting osd_memory_target on smithi131 to 4288M 2024-02-27T06:17:38.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:37 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/831531925' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "024442d9-76aa-404a-a379-cdab69e74e65"}]: dispatch 2024-02-27T06:17:38.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:37 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/831531925' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "024442d9-76aa-404a-a379-cdab69e74e65"}]': finished 2024-02-27T06:17:38.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:37 smithi131 ceph-mon[24834]: osdmap e24: 4 total, 3 up, 4 in 2024-02-27T06:17:38.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:37 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:17:38.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:37 smithi150 ceph-mon[26806]: Detected new or changed devices on smithi131 2024-02-27T06:17:38.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:37 smithi150 ceph-mon[26806]: Adjusting osd_memory_target on smithi131 to 4288M 2024-02-27T06:17:38.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:37 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/831531925' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "024442d9-76aa-404a-a379-cdab69e74e65"}]: dispatch 2024-02-27T06:17:38.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:37 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/831531925' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "024442d9-76aa-404a-a379-cdab69e74e65"}]': finished 2024-02-27T06:17:38.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:37 smithi150 ceph-mon[26806]: osdmap e24: 4 total, 3 up, 4 in 2024-02-27T06:17:38.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:37 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:17:39.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:38 smithi131 ceph-mon[24834]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:39.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:38 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/885025782' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:17:39.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:38 smithi150 ceph-mon[26806]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:39.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:38 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/885025782' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:17:41.044 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:40 smithi131 ceph-mon[24834]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-27T06:17:41.101 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:40 smithi150 ceph-mon[26806]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-27T06:17:43.046 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:42 smithi131 ceph-mon[24834]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-27T06:17:43.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:42 smithi150 ceph-mon[26806]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-27T06:17:44.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:43 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T06:17:44.096 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:43 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:44.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:43 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T06:17:44.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:43 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:45.188 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:44 smithi131 ceph-mon[24834]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-27T06:17:45.188 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:44 smithi131 ceph-mon[24834]: Deploying daemon osd.3 on smithi131 2024-02-27T06:17:45.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:44 smithi150 ceph-mon[26806]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-27T06:17:45.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:44 smithi150 ceph-mon[26806]: Deploying daemon osd.3 on smithi131 2024-02-27T06:17:47.041 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:46 smithi131 ceph-mon[24834]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-27T06:17:47.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:46 smithi150 ceph-mon[26806]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-27T06:17:48.955 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:48 smithi131 ceph-mon[24834]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 77 KiB/s wr, 5 op/s 2024-02-27T06:17:49.297 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:48 smithi150 ceph-mon[26806]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 77 KiB/s wr, 5 op/s 2024-02-27T06:17:50.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:49 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:50.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:49 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:17:50.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:49 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:50.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:49 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:17:50.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:49 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:50.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:49 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:17:50.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:49 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:50.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:49 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:17:51.261 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:50 smithi131 ceph-mon[24834]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-27T06:17:51.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:50 smithi150 ceph-mon[26806]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-27T06:17:51.955 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 3 on host 'smithi131' 2024-02-27T06:17:52.632 DEBUG:teuthology.orchestra.run.smithi131:osd.3> sudo journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.3.service 2024-02-27T06:17:52.634 INFO:tasks.cephadm:Deploying osd.4 on smithi150 with /dev/vg_nvme/lv_4... 2024-02-27T06:17:52.634 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-27T06:17:53.221 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:52 smithi150 ceph-mon[26806]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:53.221 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:52 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:53.222 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:52 smithi150 ceph-mon[26806]: from='osd.3 [v2:172.21.15.131:6826/475791189,v1:172.21.15.131:6827/475791189]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T06:17:53.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:52 smithi131 ceph-mon[24834]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:53.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:52 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:53.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:52 smithi131 ceph-mon[24834]: from='osd.3 [v2:172.21.15.131:6826/475791189,v1:172.21.15.131:6827/475791189]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T06:17:54.063 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:53 smithi150 ceph-mon[26806]: from='osd.3 [v2:172.21.15.131:6826/475791189,v1:172.21.15.131:6827/475791189]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-27T06:17:54.063 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:53 smithi150 ceph-mon[26806]: osdmap e25: 4 total, 3 up, 4 in 2024-02-27T06:17:54.064 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:53 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:17:54.064 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:53 smithi150 ceph-mon[26806]: from='osd.3 [v2:172.21.15.131:6826/475791189,v1:172.21.15.131:6827/475791189]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:17:54.064 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:53 smithi150 ceph-mon[26806]: from='osd.3 [v2:172.21.15.131:6826/475791189,v1:172.21.15.131:6827/475791189]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-02-27T06:17:54.064 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:53 smithi150 ceph-mon[26806]: osdmap e26: 4 total, 3 up, 4 in 2024-02-27T06:17:54.064 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:53 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:17:54.293 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:53 smithi131 ceph-mon[24834]: from='osd.3 [v2:172.21.15.131:6826/475791189,v1:172.21.15.131:6827/475791189]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-27T06:17:54.293 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:53 smithi131 ceph-mon[24834]: osdmap e25: 4 total, 3 up, 4 in 2024-02-27T06:17:54.293 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:53 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:17:54.293 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:53 smithi131 ceph-mon[24834]: from='osd.3 [v2:172.21.15.131:6826/475791189,v1:172.21.15.131:6827/475791189]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:17:54.293 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:53 smithi131 ceph-mon[24834]: from='osd.3 [v2:172.21.15.131:6826/475791189,v1:172.21.15.131:6827/475791189]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-02-27T06:17:54.293 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:53 smithi131 ceph-mon[24834]: osdmap e26: 4 total, 3 up, 4 in 2024-02-27T06:17:54.293 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:53 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:17:54.294 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:17:53 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[46245]: 2024-02-27T06:17:53.959+0000 7f67eb21a700 -1 osd.3 0 waiting for initial osdmap 2024-02-27T06:17:54.294 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:17:53 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[46245]: 2024-02-27T06:17:53.963+0000 7f67e63b2700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:17:54.569 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-27T06:17:54.569 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-27T06:17:54.569 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-02-27T06:17:54.570 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-02-27T06:17:54.570 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134147 s, 782 MB/s 2024-02-27T06:17:54.570 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-02-27T06:17:55.189 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_4 2024-02-27T06:17:55.213 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:54 smithi131 ceph-mon[24834]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:55.213 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:54 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:17:55.213 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:54 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:17:55.214 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:54 smithi150 ceph-mon[26806]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:55.215 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:54 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:17:55.215 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:54 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:17:56.062 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:55 smithi150 ceph-mon[26806]: purged_snaps scrub starts 2024-02-27T06:17:56.062 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:55 smithi150 ceph-mon[26806]: purged_snaps scrub ok 2024-02-27T06:17:56.062 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:55 smithi150 ceph-mon[26806]: osd.3 [v2:172.21.15.131:6826/475791189,v1:172.21.15.131:6827/475791189] boot 2024-02-27T06:17:56.062 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:55 smithi150 ceph-mon[26806]: osdmap e27: 4 total, 4 up, 4 in 2024-02-27T06:17:56.062 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:55 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:17:56.062 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:55 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:56.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:55 smithi131 ceph-mon[24834]: purged_snaps scrub starts 2024-02-27T06:17:56.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:55 smithi131 ceph-mon[24834]: purged_snaps scrub ok 2024-02-27T06:17:56.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:55 smithi131 ceph-mon[24834]: osd.3 [v2:172.21.15.131:6826/475791189,v1:172.21.15.131:6827/475791189] boot 2024-02-27T06:17:56.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:55 smithi131 ceph-mon[24834]: osdmap e27: 4 total, 4 up, 4 in 2024-02-27T06:17:56.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:55 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:17:56.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:55 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:17:56.972 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:56 smithi131 ceph-mon[24834]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:56.972 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:56 smithi131 ceph-mon[24834]: osdmap e28: 4 total, 4 up, 4 in 2024-02-27T06:17:57.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:56 smithi150 ceph-mon[26806]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-27T06:17:57.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:56 smithi150 ceph-mon[26806]: osdmap e28: 4 total, 4 up, 4 in 2024-02-27T06:17:58.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:57 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:17:58.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:57 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:17:58.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:57 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:58.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:57 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:17:58.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:57 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:17:58.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:57 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:17:59.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:58 smithi131 ceph-mon[24834]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:17:59.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:17:58 smithi131 ceph-mon[24834]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:17:59.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:58 smithi150 ceph-mon[26806]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:17:59.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:17:58 smithi150 ceph-mon[26806]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:00.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: Detected new or changed devices on smithi131 2024-02-27T06:18:00.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:00.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:00.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:00.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:00.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:00.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: Adjusting osd_memory_target on smithi131 to 3216M 2024-02-27T06:18:00.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:00.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:00.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:18:00.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:00.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:18:00.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:00.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/1915197506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c411bbd-9524-414b-b849-5831d601040a"}]: dispatch 2024-02-27T06:18:00.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c411bbd-9524-414b-b849-5831d601040a"}]: dispatch 2024-02-27T06:18:00.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c411bbd-9524-414b-b849-5831d601040a"}]': finished 2024-02-27T06:18:00.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: osdmap e29: 5 total, 4 up, 5 in 2024-02-27T06:18:00.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:18:00.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/242476681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:18:00.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:00 smithi131 ceph-mon[24834]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:00.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: Detected new or changed devices on smithi131 2024-02-27T06:18:00.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:00.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:00.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:00.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: Adjusting osd_memory_target on smithi131 to 3216M 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/1915197506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c411bbd-9524-414b-b849-5831d601040a"}]: dispatch 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c411bbd-9524-414b-b849-5831d601040a"}]: dispatch 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c411bbd-9524-414b-b849-5831d601040a"}]': finished 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: osdmap e29: 5 total, 4 up, 5 in 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/242476681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:18:00.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:00 smithi150 ceph-mon[26806]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:03.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:02 smithi131 ceph-mon[24834]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:03.098 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:02 smithi150 ceph-mon[26806]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:05.041 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:04 smithi150 ceph-mon[26806]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:05.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:04 smithi131 ceph-mon[24834]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:06.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:05 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T06:18:06.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:05 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:06.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:05 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T06:18:06.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:05 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:07.000 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:06 smithi150 ceph-mon[26806]: Deploying daemon osd.4 on smithi150 2024-02-27T06:18:07.000 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:06 smithi150 ceph-mon[26806]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:07.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:06 smithi131 ceph-mon[24834]: Deploying daemon osd.4 on smithi150 2024-02-27T06:18:07.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:06 smithi131 ceph-mon[24834]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:09.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:08 smithi150 ceph-mon[26806]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:09.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:08 smithi131 ceph-mon[24834]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:11.261 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:10 smithi150 ceph-mon[26806]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:11.261 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:11.261 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:18:11.262 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:11.262 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:18:11.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:10 smithi131 ceph-mon[24834]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:11.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:11.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:18:11.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:11.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:18:13.314 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:12 smithi150 ceph-mon[26806]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:13.346 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 4 on host 'smithi150' 2024-02-27T06:18:13.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:12 smithi131 ceph-mon[24834]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:13.910 DEBUG:teuthology.orchestra.run.smithi150:osd.4> sudo journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.4.service 2024-02-27T06:18:13.953 INFO:tasks.cephadm:Deploying osd.5 on smithi150 with /dev/vg_nvme/lv_3... 2024-02-27T06:18:13.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-27T06:18:14.239 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:13 smithi150 ceph-mon[26806]: from='osd.4 [v2:172.21.15.150:6800/3033573590,v1:172.21.15.150:6801/3033573590]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T06:18:14.239 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:13 smithi150 ceph-mon[26806]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T06:18:14.239 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:13 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:14.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:13 smithi131 ceph-mon[24834]: from='osd.4 [v2:172.21.15.150:6800/3033573590,v1:172.21.15.150:6801/3033573590]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T06:18:14.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:13 smithi131 ceph-mon[24834]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T06:18:14.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:13 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:15.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:14 smithi150 ceph-mon[26806]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:15.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:14 smithi150 ceph-mon[26806]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-27T06:18:15.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:14 smithi150 ceph-mon[26806]: osdmap e30: 5 total, 4 up, 5 in 2024-02-27T06:18:15.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:14 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:18:15.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:14 smithi150 ceph-mon[26806]: from='osd.4 [v2:172.21.15.150:6800/3033573590,v1:172.21.15.150:6801/3033573590]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:18:15.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:14 smithi150 ceph-mon[26806]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:18:15.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:14 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:15.103 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:18:14 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[30957]: 2024-02-27T06:18:14.956+0000 7f4b38a39700 -1 osd.4 0 waiting for initial osdmap 2024-02-27T06:18:15.104 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:18:14 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[30957]: 2024-02-27T06:18:14.961+0000 7f4b313cc700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:18:15.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:14 smithi131 ceph-mon[24834]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:15.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:14 smithi131 ceph-mon[24834]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-27T06:18:15.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:14 smithi131 ceph-mon[24834]: osdmap e30: 5 total, 4 up, 5 in 2024-02-27T06:18:15.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:14 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:18:15.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:14 smithi131 ceph-mon[24834]: from='osd.4 [v2:172.21.15.150:6800/3033573590,v1:172.21.15.150:6801/3033573590]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:18:15.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:14 smithi131 ceph-mon[24834]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:18:15.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:14 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-27T06:18:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-27T06:18:15.876 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-02-27T06:18:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-02-27T06:18:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012755 s, 822 MB/s 2024-02-27T06:18:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-02-27T06:18:16.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:15 smithi150 ceph-mon[26806]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-27T06:18:16.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:15 smithi150 ceph-mon[26806]: osdmap e31: 5 total, 4 up, 5 in 2024-02-27T06:18:16.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:15 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:18:16.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:15 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:18:16.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:15 smithi131 ceph-mon[24834]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-27T06:18:16.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:15 smithi131 ceph-mon[24834]: osdmap e31: 5 total, 4 up, 5 in 2024-02-27T06:18:16.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:15 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:18:16.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:15 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:18:16.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_3 2024-02-27T06:18:17.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:16 smithi150 ceph-mon[26806]: purged_snaps scrub starts 2024-02-27T06:18:17.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:16 smithi150 ceph-mon[26806]: purged_snaps scrub ok 2024-02-27T06:18:17.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:16 smithi150 ceph-mon[26806]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:17.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:16 smithi150 ceph-mon[26806]: osd.4 [v2:172.21.15.150:6800/3033573590,v1:172.21.15.150:6801/3033573590] boot 2024-02-27T06:18:17.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:16 smithi150 ceph-mon[26806]: osdmap e32: 5 total, 5 up, 5 in 2024-02-27T06:18:17.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:16 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:18:17.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:16 smithi150 ceph-mon[26806]: osdmap e33: 5 total, 5 up, 5 in 2024-02-27T06:18:17.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:16 smithi131 ceph-mon[24834]: purged_snaps scrub starts 2024-02-27T06:18:17.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:16 smithi131 ceph-mon[24834]: purged_snaps scrub ok 2024-02-27T06:18:17.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:16 smithi131 ceph-mon[24834]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-27T06:18:17.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:16 smithi131 ceph-mon[24834]: osd.4 [v2:172.21.15.150:6800/3033573590,v1:172.21.15.150:6801/3033573590] boot 2024-02-27T06:18:17.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:16 smithi131 ceph-mon[24834]: osdmap e32: 5 total, 5 up, 5 in 2024-02-27T06:18:17.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:16 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:18:17.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:16 smithi131 ceph-mon[24834]: osdmap e33: 5 total, 5 up, 5 in 2024-02-27T06:18:19.076 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:18 smithi150 ceph-mon[26806]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:19.076 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:18 smithi150 ceph-mon[26806]: osdmap e34: 5 total, 5 up, 5 in 2024-02-27T06:18:19.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:18 smithi131 ceph-mon[24834]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:19.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:18 smithi131 ceph-mon[24834]: osdmap e34: 5 total, 5 up, 5 in 2024-02-27T06:18:20.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:19 smithi131 ceph-mon[24834]: osdmap e35: 5 total, 5 up, 5 in 2024-02-27T06:18:20.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:19 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:20.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:19 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:20.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:19 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:20.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:19 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:20.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:19 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:18:20.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:19 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:20.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:19 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:18:20.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:19 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:20.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:19 smithi150 ceph-mon[26806]: osdmap e35: 5 total, 5 up, 5 in 2024-02-27T06:18:20.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:19 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:20.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:19 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:20.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:19 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:20.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:19 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:20.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:19 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:18:20.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:19 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:20.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:19 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:18:20.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:19 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:21.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:20 smithi150 ceph-mon[26806]: Detected new or changed devices on smithi150 2024-02-27T06:18:21.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:20 smithi150 ceph-mon[26806]: Adjusting osd_memory_target on smithi150 to 15936M 2024-02-27T06:18:21.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:20 smithi150 ceph-mon[26806]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-27T06:18:21.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:20 smithi150 ceph-mon[26806]: osdmap e36: 5 total, 5 up, 5 in 2024-02-27T06:18:21.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:20 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:18:21.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:20 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:18:21.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:20 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:21.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:20 smithi131 ceph-mon[24834]: Detected new or changed devices on smithi150 2024-02-27T06:18:21.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:20 smithi131 ceph-mon[24834]: Adjusting osd_memory_target on smithi150 to 15936M 2024-02-27T06:18:21.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:20 smithi131 ceph-mon[24834]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-27T06:18:21.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:20 smithi131 ceph-mon[24834]: osdmap e36: 5 total, 5 up, 5 in 2024-02-27T06:18:21.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:20 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:18:21.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:20 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:18:21.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:20 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:22.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:21 smithi150 ceph-mon[26806]: from='client.24153 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:18:22.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:21 smithi131 ceph-mon[24834]: from='client.24153 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:18:23.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:22 smithi150 ceph-mon[26806]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-02-27T06:18:23.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:22 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/2407983746' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4cd7061-7b4a-46ab-89a2-e5c2c32d725e"}]: dispatch 2024-02-27T06:18:23.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:22 smithi150 ceph-mon[26806]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4cd7061-7b4a-46ab-89a2-e5c2c32d725e"}]: dispatch 2024-02-27T06:18:23.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:22 smithi150 ceph-mon[26806]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4cd7061-7b4a-46ab-89a2-e5c2c32d725e"}]': finished 2024-02-27T06:18:23.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:22 smithi150 ceph-mon[26806]: osdmap e37: 6 total, 5 up, 6 in 2024-02-27T06:18:23.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:18:23.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:22 smithi131 ceph-mon[24834]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-02-27T06:18:23.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:22 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/2407983746' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4cd7061-7b4a-46ab-89a2-e5c2c32d725e"}]: dispatch 2024-02-27T06:18:23.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:22 smithi131 ceph-mon[24834]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4cd7061-7b4a-46ab-89a2-e5c2c32d725e"}]: dispatch 2024-02-27T06:18:23.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:22 smithi131 ceph-mon[24834]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4cd7061-7b4a-46ab-89a2-e5c2c32d725e"}]': finished 2024-02-27T06:18:23.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:22 smithi131 ceph-mon[24834]: osdmap e37: 6 total, 5 up, 6 in 2024-02-27T06:18:23.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:18:24.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:23 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/3224852242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:18:24.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:23 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/3224852242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:18:25.315 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:24 smithi150 ceph-mon[26806]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-27T06:18:25.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:24 smithi131 ceph-mon[24834]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-27T06:18:26.155 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:18:26.155 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:18:26.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:18:26.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:18:27.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:26 smithi150 ceph-mon[26806]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:27.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:26 smithi131 ceph-mon[24834]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:29.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:28 smithi150 ceph-mon[26806]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:29.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:28 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T06:18:29.102 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:28 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:29.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:28 smithi131 ceph-mon[24834]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:29.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:28 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T06:18:29.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:28 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:30.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:29 smithi131 ceph-mon[24834]: Deploying daemon osd.5 on smithi150 2024-02-27T06:18:30.191 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:29 smithi150 ceph-mon[26806]: Deploying daemon osd.5 on smithi150 2024-02-27T06:18:31.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:30 smithi150 ceph-mon[26806]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:31.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:30 smithi131 ceph-mon[24834]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:33.088 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:32 smithi150 ceph-mon[26806]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:33.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:32 smithi131 ceph-mon[24834]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:35.274 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:34 smithi150 ceph-mon[26806]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:35.274 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:35.274 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:18:35.274 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:35.275 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:34 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:18:35.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:34 smithi131 ceph-mon[24834]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:35.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:35.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:18:35.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:35.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:34 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:18:36.944 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 5 on host 'smithi150' 2024-02-27T06:18:37.189 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:36 smithi150 ceph-mon[26806]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:37.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:36 smithi131 ceph-mon[24834]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:37.800 DEBUG:teuthology.orchestra.run.smithi150:osd.5> sudo journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.5.service 2024-02-27T06:18:37.803 INFO:tasks.cephadm:Deploying osd.6 on smithi150 with /dev/vg_nvme/lv_2... 2024-02-27T06:18:37.803 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-27T06:18:38.099 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:37 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:38.099 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:37 smithi150 ceph-mon[26806]: from='osd.5 [v2:172.21.15.150:6808/3851715073,v1:172.21.15.150:6809/3851715073]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T06:18:38.099 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:37 smithi150 ceph-mon[26806]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T06:18:38.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:37 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:38.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:37 smithi131 ceph-mon[24834]: from='osd.5 [v2:172.21.15.150:6808/3851715073,v1:172.21.15.150:6809/3851715073]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T06:18:38.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:37 smithi131 ceph-mon[24834]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T06:18:39.338 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:38 smithi150 ceph-mon[26806]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:39.338 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:38 smithi150 ceph-mon[26806]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-27T06:18:39.338 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:38 smithi150 ceph-mon[26806]: osdmap e38: 6 total, 5 up, 6 in 2024-02-27T06:18:39.338 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:38 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:18:39.339 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:38 smithi150 ceph-mon[26806]: from='osd.5 [v2:172.21.15.150:6808/3851715073,v1:172.21.15.150:6809/3851715073]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:18:39.339 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:38 smithi150 ceph-mon[26806]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:18:39.339 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:38 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:39.339 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:18:38 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[33943]: 2024-02-27T06:18:38.953+0000 7fd4d728d700 -1 osd.5 0 waiting for initial osdmap 2024-02-27T06:18:39.339 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:18:38 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[33943]: 2024-02-27T06:18:38.959+0000 7fd4d2425700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:18:39.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:38 smithi131 ceph-mon[24834]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:39.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:38 smithi131 ceph-mon[24834]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-27T06:18:39.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:38 smithi131 ceph-mon[24834]: osdmap e38: 6 total, 5 up, 6 in 2024-02-27T06:18:39.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:38 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:18:39.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:38 smithi131 ceph-mon[24834]: from='osd.5 [v2:172.21.15.150:6808/3851715073,v1:172.21.15.150:6809/3851715073]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:18:39.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:38 smithi131 ceph-mon[24834]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:18:39.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:38 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:40.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:39 smithi131 ceph-mon[24834]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-27T06:18:40.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:39 smithi131 ceph-mon[24834]: osdmap e39: 6 total, 5 up, 6 in 2024-02-27T06:18:40.097 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:39 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:18:40.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:39 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:18:40.172 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:39 smithi150 ceph-mon[26806]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-27T06:18:40.172 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:39 smithi150 ceph-mon[26806]: osdmap e39: 6 total, 5 up, 6 in 2024-02-27T06:18:40.172 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:39 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:18:40.172 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:39 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:18:40.374 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-27T06:18:40.374 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-27T06:18:40.374 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-02-27T06:18:40.374 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-02-27T06:18:40.374 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125 s, 839 MB/s 2024-02-27T06:18:40.374 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-02-27T06:18:41.015 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_2 2024-02-27T06:18:41.259 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:40 smithi150 ceph-mon[26806]: purged_snaps scrub starts 2024-02-27T06:18:41.259 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:40 smithi150 ceph-mon[26806]: purged_snaps scrub ok 2024-02-27T06:18:41.259 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:40 smithi150 ceph-mon[26806]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:41.259 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:40 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:18:41.259 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:40 smithi150 ceph-mon[26806]: osd.5 [v2:172.21.15.150:6808/3851715073,v1:172.21.15.150:6809/3851715073] boot 2024-02-27T06:18:41.259 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:40 smithi150 ceph-mon[26806]: osdmap e40: 6 total, 6 up, 6 in 2024-02-27T06:18:41.259 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:40 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:18:41.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:40 smithi131 ceph-mon[24834]: purged_snaps scrub starts 2024-02-27T06:18:41.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:40 smithi131 ceph-mon[24834]: purged_snaps scrub ok 2024-02-27T06:18:41.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:40 smithi131 ceph-mon[24834]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-27T06:18:41.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:40 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:18:41.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:40 smithi131 ceph-mon[24834]: osd.5 [v2:172.21.15.150:6808/3851715073,v1:172.21.15.150:6809/3851715073] boot 2024-02-27T06:18:41.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:40 smithi131 ceph-mon[24834]: osdmap e40: 6 total, 6 up, 6 in 2024-02-27T06:18:41.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:40 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:18:42.056 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:41 smithi150 ceph-mon[26806]: osdmap e41: 6 total, 6 up, 6 in 2024-02-27T06:18:42.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:41 smithi131 ceph-mon[24834]: osdmap e41: 6 total, 6 up, 6 in 2024-02-27T06:18:43.259 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:42 smithi150 ceph-mon[26806]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:43.259 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:42 smithi150 ceph-mon[26806]: osdmap e42: 6 total, 6 up, 6 in 2024-02-27T06:18:43.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:42 smithi131 ceph-mon[24834]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:43.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:42 smithi131 ceph-mon[24834]: osdmap e42: 6 total, 6 up, 6 in 2024-02-27T06:18:44.091 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:43 smithi150 ceph-mon[26806]: osdmap e43: 6 total, 6 up, 6 in 2024-02-27T06:18:44.091 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:43 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:44.091 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:43 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:44.091 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:43 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:44.091 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:43 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:44.091 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:43 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:44.091 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:43 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:18:44.092 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:43 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:44.092 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:43 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:18:44.092 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:43 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:44.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:43 smithi131 ceph-mon[24834]: osdmap e43: 6 total, 6 up, 6 in 2024-02-27T06:18:44.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:43 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:44.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:43 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:44.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:43 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:18:44.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:43 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:44.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:43 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:44.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:43 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:18:44.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:43 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:44.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:43 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:18:44.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:43 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:45.223 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:44 smithi150 ceph-mon[26806]: Detected new or changed devices on smithi150 2024-02-27T06:18:45.223 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:44 smithi150 ceph-mon[26806]: Adjusting osd_memory_target on smithi150 to 7968M 2024-02-27T06:18:45.223 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:44 smithi150 ceph-mon[26806]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:45.223 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:44 smithi150 ceph-mon[26806]: osdmap e44: 6 total, 6 up, 6 in 2024-02-27T06:18:45.223 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:44 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:18:45.224 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:44 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:18:45.224 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:44 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:45.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:44 smithi131 ceph-mon[24834]: Detected new or changed devices on smithi150 2024-02-27T06:18:45.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:44 smithi131 ceph-mon[24834]: Adjusting osd_memory_target on smithi150 to 7968M 2024-02-27T06:18:45.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:44 smithi131 ceph-mon[24834]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:45.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:44 smithi131 ceph-mon[24834]: osdmap e44: 6 total, 6 up, 6 in 2024-02-27T06:18:45.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:44 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:18:45.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:44 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:18:45.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:44 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:46.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:45 smithi150 ceph-mon[26806]: from='client.24171 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:18:46.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:45 smithi131 ceph-mon[24834]: from='client.24171 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:18:47.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:46 smithi131 ceph-mon[24834]: pgmap v132: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:47.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:46 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/918131091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab5fe169-f1c9-4241-899e-fc34b5ad278d"}]: dispatch 2024-02-27T06:18:47.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:46 smithi131 ceph-mon[24834]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab5fe169-f1c9-4241-899e-fc34b5ad278d"}]: dispatch 2024-02-27T06:18:47.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:46 smithi131 ceph-mon[24834]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab5fe169-f1c9-4241-899e-fc34b5ad278d"}]': finished 2024-02-27T06:18:47.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:46 smithi131 ceph-mon[24834]: osdmap e45: 7 total, 6 up, 7 in 2024-02-27T06:18:47.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:46 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:18:47.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:46 smithi150 ceph-mon[26806]: pgmap v132: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:47.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:46 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/918131091' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab5fe169-f1c9-4241-899e-fc34b5ad278d"}]: dispatch 2024-02-27T06:18:47.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:46 smithi150 ceph-mon[26806]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab5fe169-f1c9-4241-899e-fc34b5ad278d"}]: dispatch 2024-02-27T06:18:47.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:46 smithi150 ceph-mon[26806]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab5fe169-f1c9-4241-899e-fc34b5ad278d"}]': finished 2024-02-27T06:18:47.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:46 smithi150 ceph-mon[26806]: osdmap e45: 7 total, 6 up, 7 in 2024-02-27T06:18:47.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:46 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:18:48.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:47 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/1679919338' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:18:48.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:47 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/1679919338' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:18:49.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:48 smithi131 ceph-mon[24834]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:49.352 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:48 smithi150 ceph-mon[26806]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:50.251 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:49 smithi150 ceph-mon[26806]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:50.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:49 smithi131 ceph-mon[24834]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:52.948 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:52 smithi150 ceph-mon[26806]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:53.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:52 smithi131 ceph-mon[24834]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:54.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:53 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T06:18:54.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:53 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:54.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:53 smithi150 ceph-mon[26806]: Deploying daemon osd.6 on smithi150 2024-02-27T06:18:54.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:53 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T06:18:54.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:53 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:54.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:53 smithi131 ceph-mon[24834]: Deploying daemon osd.6 on smithi150 2024-02-27T06:18:55.335 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:54 smithi150 ceph-mon[26806]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:55.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:54 smithi131 ceph-mon[24834]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:57.043 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:56 smithi150 ceph-mon[26806]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:57.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:56 smithi131 ceph-mon[24834]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:59.229 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:58 smithi150 ceph-mon[26806]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:59.229 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:58 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:59.229 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:58 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:18:59.229 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:58 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:59.229 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:18:58 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:18:59.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:58 smithi131 ceph-mon[24834]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:18:59.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:58 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:18:59.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:58 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:18:59.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:58 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:18:59.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:18:58 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:19:00.954 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:00 smithi150 ceph-mon[26806]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:19:01.011 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 6 on host 'smithi150' 2024-02-27T06:19:01.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:00 smithi131 ceph-mon[24834]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:19:01.598 DEBUG:teuthology.orchestra.run.smithi150:osd.6> sudo journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.6.service 2024-02-27T06:19:01.601 INFO:tasks.cephadm:Deploying osd.7 on smithi150 with /dev/vg_nvme/lv_1... 2024-02-27T06:19:01.601 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-27T06:19:02.202 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:01 smithi150 ceph-mon[26806]: from='osd.6 [v2:172.21.15.150:6816/2464259658,v1:172.21.15.150:6817/2464259658]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T06:19:02.202 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:01 smithi150 ceph-mon[26806]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T06:19:02.202 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:01 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:02.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:01 smithi131 ceph-mon[24834]: from='osd.6 [v2:172.21.15.150:6816/2464259658,v1:172.21.15.150:6817/2464259658]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T06:19:02.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:01 smithi131 ceph-mon[24834]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T06:19:02.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:01 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:03.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:02 smithi150 ceph-mon[26806]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:19:03.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:02 smithi150 ceph-mon[26806]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-27T06:19:03.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:02 smithi150 ceph-mon[26806]: osdmap e46: 7 total, 6 up, 7 in 2024-02-27T06:19:03.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:02 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:19:03.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:02 smithi150 ceph-mon[26806]: from='osd.6 [v2:172.21.15.150:6816/2464259658,v1:172.21.15.150:6817/2464259658]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:19:03.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:02 smithi150 ceph-mon[26806]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:19:03.104 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:19:02 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[36932]: 2024-02-27T06:19:02.879+0000 7f844323e700 -1 osd.6 0 waiting for initial osdmap 2024-02-27T06:19:03.104 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:19:02 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[36932]: 2024-02-27T06:19:02.886+0000 7f843dbd5700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:19:03.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:02 smithi131 ceph-mon[24834]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:19:03.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:02 smithi131 ceph-mon[24834]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-27T06:19:03.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:02 smithi131 ceph-mon[24834]: osdmap e46: 7 total, 6 up, 7 in 2024-02-27T06:19:03.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:02 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:19:03.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:02 smithi131 ceph-mon[24834]: from='osd.6 [v2:172.21.15.150:6816/2464259658,v1:172.21.15.150:6817/2464259658]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:19:03.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:02 smithi131 ceph-mon[24834]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:19:04.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:03 smithi150 ceph-mon[26806]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-27T06:19:04.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:03 smithi150 ceph-mon[26806]: osdmap e47: 7 total, 6 up, 7 in 2024-02-27T06:19:04.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:03 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:19:04.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:03 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:19:04.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:03 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:04.297 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-27T06:19:04.298 INFO:teuthology.orchestra.run.smithi150.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-27T06:19:04.298 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10+0 records in 2024-02-27T06:19:04.298 INFO:teuthology.orchestra.run.smithi150.stderr:10+0 records out 2024-02-27T06:19:04.298 INFO:teuthology.orchestra.run.smithi150.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134836 s, 778 MB/s 2024-02-27T06:19:04.298 INFO:teuthology.orchestra.run.smithi150.stderr:--> Zapping successful for: 2024-02-27T06:19:04.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:03 smithi131 ceph-mon[24834]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-27T06:19:04.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:03 smithi131 ceph-mon[24834]: osdmap e47: 7 total, 6 up, 7 in 2024-02-27T06:19:04.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:03 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:19:04.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:03 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:19:04.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:03 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:05.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:04 smithi131 ceph-mon[24834]: purged_snaps scrub starts 2024-02-27T06:19:05.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:04 smithi131 ceph-mon[24834]: purged_snaps scrub ok 2024-02-27T06:19:05.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:04 smithi131 ceph-mon[24834]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:19:05.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:04 smithi131 ceph-mon[24834]: osd.6 [v2:172.21.15.150:6816/2464259658,v1:172.21.15.150:6817/2464259658] boot 2024-02-27T06:19:05.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:04 smithi131 ceph-mon[24834]: osdmap e48: 7 total, 7 up, 7 in 2024-02-27T06:19:05.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:04 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:19:05.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:04 smithi150 ceph-mon[26806]: purged_snaps scrub starts 2024-02-27T06:19:05.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:04 smithi150 ceph-mon[26806]: purged_snaps scrub ok 2024-02-27T06:19:05.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:04 smithi150 ceph-mon[26806]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-27T06:19:05.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:04 smithi150 ceph-mon[26806]: osd.6 [v2:172.21.15.150:6816/2464259658,v1:172.21.15.150:6817/2464259658] boot 2024-02-27T06:19:05.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:04 smithi150 ceph-mon[26806]: osdmap e48: 7 total, 7 up, 7 in 2024-02-27T06:19:05.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:04 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:19:05.814 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch daemon add osd smithi150:vg_nvme/lv_1 2024-02-27T06:19:06.042 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:05 smithi150 ceph-mon[26806]: osdmap e49: 7 total, 7 up, 7 in 2024-02-27T06:19:06.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:05 smithi131 ceph-mon[24834]: osdmap e49: 7 total, 7 up, 7 in 2024-02-27T06:19:07.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:06 smithi150 ceph-mon[26806]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:07.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:06 smithi150 ceph-mon[26806]: osdmap e50: 7 total, 7 up, 7 in 2024-02-27T06:19:07.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:06 smithi150 ceph-mon[26806]: osdmap e51: 7 total, 7 up, 7 in 2024-02-27T06:19:07.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:06 smithi131 ceph-mon[24834]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:07.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:06 smithi131 ceph-mon[24834]: osdmap e50: 7 total, 7 up, 7 in 2024-02-27T06:19:07.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:06 smithi131 ceph-mon[24834]: osdmap e51: 7 total, 7 up, 7 in 2024-02-27T06:19:08.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: Detected new or changed devices on smithi150 2024-02-27T06:19:08.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:08.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:08.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:08.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:08.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: Adjusting osd_memory_target on smithi150 to 5312M 2024-02-27T06:19:08.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:08.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:08.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:19:08.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:19:08.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:19:08.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:08.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:08.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:08 smithi131 ceph-mon[24834]: osdmap e52: 7 total, 7 up, 7 in 2024-02-27T06:19:08.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: Detected new or changed devices on smithi150 2024-02-27T06:19:08.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:08.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:08.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:08.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:08.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: Adjusting osd_memory_target on smithi150 to 5312M 2024-02-27T06:19:08.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:08.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:08.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:19:08.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:19:08.854 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:19:08.854 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:08.854 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:08.854 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:08 smithi150 ceph-mon[26806]: osdmap e52: 7 total, 7 up, 7 in 2024-02-27T06:19:09.722 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:09 smithi131 ceph-mon[24834]: from='client.24189 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:19:09.722 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:09 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:19:09.723 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:09 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:19:09.723 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:09 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:19:09.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:09 smithi150 ceph-mon[26806]: from='client.24189 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi150:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:19:09.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:09 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-27T06:19:09.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:09 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-27T06:19:09.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:09 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:19:10.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:10 smithi131 ceph-mon[24834]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:10.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:10 smithi150 ceph-mon[26806]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:11.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:11 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/2744335396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d01488da-c749-4e79-9c3a-9e66e5a50f48"}]: dispatch 2024-02-27T06:19:11.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:11 smithi131 ceph-mon[24834]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d01488da-c749-4e79-9c3a-9e66e5a50f48"}]: dispatch 2024-02-27T06:19:11.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:11 smithi131 ceph-mon[24834]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d01488da-c749-4e79-9c3a-9e66e5a50f48"}]': finished 2024-02-27T06:19:11.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:11 smithi131 ceph-mon[24834]: osdmap e53: 8 total, 7 up, 8 in 2024-02-27T06:19:11.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:11 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:19:11.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:11 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/2744335396' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d01488da-c749-4e79-9c3a-9e66e5a50f48"}]: dispatch 2024-02-27T06:19:11.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:11 smithi150 ceph-mon[26806]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d01488da-c749-4e79-9c3a-9e66e5a50f48"}]: dispatch 2024-02-27T06:19:11.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:11 smithi150 ceph-mon[26806]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d01488da-c749-4e79-9c3a-9e66e5a50f48"}]': finished 2024-02-27T06:19:11.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:11 smithi150 ceph-mon[26806]: osdmap e53: 8 total, 7 up, 8 in 2024-02-27T06:19:11.853 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:11 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:19:12.754 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:12 smithi150 ceph-mon[26806]: from='client.? 172.21.15.150:0/2472644677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:19:12.754 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:12 smithi150 ceph-mon[26806]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-27T06:19:12.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:12 smithi131 ceph-mon[24834]: from='client.? 172.21.15.150:0/2472644677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-27T06:19:12.848 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:12 smithi131 ceph-mon[24834]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-27T06:19:15.085 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:14 smithi150 ceph-mon[26806]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-27T06:19:15.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:14 smithi131 ceph-mon[24834]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-27T06:19:17.155 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:16 smithi150 ceph-mon[26806]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-27T06:19:17.347 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:16 smithi131 ceph-mon[24834]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-27T06:19:18.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:17 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T06:19:18.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:17 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:19:18.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:17 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T06:19:18.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:17 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:19:19.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:18 smithi150 ceph-mon[26806]: Deploying daemon osd.7 on smithi150 2024-02-27T06:19:19.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:18 smithi150 ceph-mon[26806]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-27T06:19:19.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:18 smithi131 ceph-mon[24834]: Deploying daemon osd.7 on smithi150 2024-02-27T06:19:19.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:18 smithi131 ceph-mon[24834]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-27T06:19:21.080 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:20 smithi150 ceph-mon[26806]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-27T06:19:21.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:20 smithi131 ceph-mon[24834]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-27T06:19:22.928 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:22 smithi150 ceph-mon[26806]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:22.928 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:22.928 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:19:22.928 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:19:22.928 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:19:23.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:22 smithi131 ceph-mon[24834]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:23.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:23.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:19:23.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:19:23.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:19:25.135 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:24 smithi150 ceph-mon[26806]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:25.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:24 smithi131 ceph-mon[24834]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:25.812 INFO:teuthology.orchestra.run.smithi150.stdout:Created osd(s) 7 on host 'smithi150' 2024-02-27T06:19:26.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:25 smithi150 ceph-mon[26806]: from='osd.7 [v2:172.21.15.150:6824/4201152942,v1:172.21.15.150:6825/4201152942]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T06:19:26.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:25 smithi150 ceph-mon[26806]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T06:19:26.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:26.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:19:26.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:19:26.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:25 smithi131 ceph-mon[24834]: from='osd.7 [v2:172.21.15.150:6824/4201152942,v1:172.21.15.150:6825/4201152942]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T06:19:26.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:25 smithi131 ceph-mon[24834]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T06:19:26.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:26.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:19:26.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:19:26.757 DEBUG:teuthology.orchestra.run.smithi150:osd.7> sudo journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.7.service 2024-02-27T06:19:26.760 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-27T06:19:26.760 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd stat -f json 2024-02-27T06:19:27.080 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:26 smithi131 ceph-mon[24834]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:27.080 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:26 smithi131 ceph-mon[24834]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-27T06:19:27.080 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:26 smithi131 ceph-mon[24834]: osdmap e54: 8 total, 7 up, 8 in 2024-02-27T06:19:27.080 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:19:27.080 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:26 smithi131 ceph-mon[24834]: from='osd.7 [v2:172.21.15.150:6824/4201152942,v1:172.21.15.150:6825/4201152942]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:19:27.081 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:26 smithi131 ceph-mon[24834]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:19:27.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:26 smithi150 ceph-mon[26806]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:27.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:26 smithi150 ceph-mon[26806]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-27T06:19:27.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:26 smithi150 ceph-mon[26806]: osdmap e54: 8 total, 7 up, 8 in 2024-02-27T06:19:27.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:19:27.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:26 smithi150 ceph-mon[26806]: from='osd.7 [v2:172.21.15.150:6824/4201152942,v1:172.21.15.150:6825/4201152942]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:19:27.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:26 smithi150 ceph-mon[26806]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:19:27.103 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:19:26 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[40007]: 2024-02-27T06:19:26.891+0000 7f6e0d7f1700 -1 osd.7 0 waiting for initial osdmap 2024-02-27T06:19:27.104 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:19:26 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[40007]: 2024-02-27T06:19:26.898+0000 7f6e08188700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:19:28.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:27 smithi150 ceph-mon[26806]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-27T06:19:28.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:27 smithi150 ceph-mon[26806]: osdmap e55: 8 total, 7 up, 8 in 2024-02-27T06:19:28.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:27 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:19:28.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:27 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:19:28.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:27 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:28.203 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:27 smithi131 ceph-mon[24834]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]': finished 2024-02-27T06:19:28.203 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:27 smithi131 ceph-mon[24834]: osdmap e55: 8 total, 7 up, 8 in 2024-02-27T06:19:28.204 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:27 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:19:28.204 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:27 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:19:28.204 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:27 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:28.899 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:19:29.168 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:28 smithi150 ceph-mon[26806]: purged_snaps scrub starts 2024-02-27T06:19:29.168 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:28 smithi150 ceph-mon[26806]: purged_snaps scrub ok 2024-02-27T06:19:29.169 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:28 smithi150 ceph-mon[26806]: pgmap v164: 1 pgs: 1 peering; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:29.169 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:28 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:19:29.169 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:28 smithi150 ceph-mon[26806]: osd.7 [v2:172.21.15.150:6824/4201152942,v1:172.21.15.150:6825/4201152942] boot 2024-02-27T06:19:29.169 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:28 smithi150 ceph-mon[26806]: osdmap e56: 8 total, 8 up, 8 in 2024-02-27T06:19:29.169 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:28 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:19:29.193 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:28 smithi131 ceph-mon[24834]: purged_snaps scrub starts 2024-02-27T06:19:29.193 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:28 smithi131 ceph-mon[24834]: purged_snaps scrub ok 2024-02-27T06:19:29.193 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:28 smithi131 ceph-mon[24834]: pgmap v164: 1 pgs: 1 peering; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2024-02-27T06:19:29.193 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:28 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:19:29.193 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:28 smithi131 ceph-mon[24834]: osd.7 [v2:172.21.15.150:6824/4201152942,v1:172.21.15.150:6825/4201152942] boot 2024-02-27T06:19:29.193 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:28 smithi131 ceph-mon[24834]: osdmap e56: 8 total, 8 up, 8 in 2024-02-27T06:19:29.193 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:28 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:19:29.609 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1709014767,"num_in_osds":8,"osd_in_since":1709014750,"num_remapped_pgs":1} 2024-02-27T06:19:29.610 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd dump --format=json 2024-02-27T06:19:29.841 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:30.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: osdmap e57: 8 total, 8 up, 8 in 2024-02-27T06:19:30.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/2667643857' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-27T06:19:30.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:30.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:30.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:30.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:30.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:30.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:30.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:30.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:19:30.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:19:30.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:19:30.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:30.189 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: osdmap e57: 8 total, 8 up, 8 in 2024-02-27T06:19:30.189 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2667643857' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-27T06:19:30.189 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:30.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:30.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:30.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:30.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:19:30.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:30.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:30.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:19:30.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:19:30.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:19:30.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:19:31.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:30 smithi131 ceph-mon[24834]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:31.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:30 smithi131 ceph-mon[24834]: Detected new or changed devices on smithi150 2024-02-27T06:19:31.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:30 smithi131 ceph-mon[24834]: Adjusting osd_memory_target on smithi150 to 3984M 2024-02-27T06:19:31.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:30 smithi131 ceph-mon[24834]: osdmap e58: 8 total, 8 up, 8 in 2024-02-27T06:19:31.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:30 smithi150 ceph-mon[26806]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:31.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:30 smithi150 ceph-mon[26806]: Detected new or changed devices on smithi150 2024-02-27T06:19:31.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:30 smithi150 ceph-mon[26806]: Adjusting osd_memory_target on smithi150 to 3984M 2024-02-27T06:19:31.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:30 smithi150 ceph-mon[26806]: osdmap e58: 8 total, 8 up, 8 in 2024-02-27T06:19:31.486 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:19:31.486 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":59,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","created":"2024-02-27T06:13:42.247590+0000","modified":"2024-02-27T06:19:30.899248+0000","last_up_change":"2024-02-27T06:19:27.889367+0000","last_in_change":"2024-02-27T06:19:10.912604+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-27T06:17:30.076310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff509ee5-7059-4672-b377-c3eb04d10814","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":3100254707},{"type":"v1","addr":"172.21.15.131:6803","nonce":3100254707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":3100254707},{"type":"v1","addr":"172.21.15.131:6805","nonce":3100254707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3100254707},{"type":"v1","addr":"172.21.15.131:6809","nonce":3100254707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":3100254707},{"type":"v1","addr":"172.21.15.131:6807","nonce":3100254707}]},"public_addr":"172.21.15.131:6803/3100254707","cluster_addr":"172.21.15.131:6805/3100254707","heartbeat_back_addr":"172.21.15.131:6809/3100254707","heartbeat_front_addr":"172.21.15.131:6807/3100254707","state":["exists","up"]},{"osd":1,"uuid":"f3aa96ce-7431-4a6f-adca-92a510c205f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1508562642},{"type":"v1","addr":"172.21.15.131:6811","nonce":1508562642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1508562642},{"type":"v1","addr":"172.21.15.131:6813","nonce":1508562642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":1508562642},{"type":"v1","addr":"172.21.15.131:6817","nonce":1508562642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1508562642},{"type":"v1","addr":"172.21.15.131:6815","nonce":1508562642}]},"public_addr":"172.21.15.131:6811/1508562642","cluster_addr":"172.21.15.131:6813/1508562642","heartbeat_back_addr":"172.21.15.131:6817/1508562642","heartbeat_front_addr":"172.21.15.131:6815/1508562642","state":["exists","up"]},{"osd":2,"uuid":"6fb48b20-5ecd-4d7c-8784-b0fea58df271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":567909672},{"type":"v1","addr":"172.21.15.131:6819","nonce":567909672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":567909672},{"type":"v1","addr":"172.21.15.131:6821","nonce":567909672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":567909672},{"type":"v1","addr":"172.21.15.131:6825","nonce":567909672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":567909672},{"type":"v1","addr":"172.21.15.131:6823","nonce":567909672}]},"public_addr":"172.21.15.131:6819/567909672","cluster_addr":"172.21.15.131:6821/567909672","heartbeat_back_addr":"172.21.15.131:6825/567909672","heartbeat_front_addr":"172.21.15.131:6823/567909672","state":["exists","up"]},{"osd":3,"uuid":"024442d9-76aa-404a-a379-cdab69e74e65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":475791189},{"type":"v1","addr":"172.21.15.131:6827","nonce":475791189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":475791189},{"type":"v1","addr":"172.21.15.131:6829","nonce":475791189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":475791189},{"type":"v1","addr":"172.21.15.131:6833","nonce":475791189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":475791189},{"type":"v1","addr":"172.21.15.131:6831","nonce":475791189}]},"public_addr":"172.21.15.131:6827/475791189","cluster_addr":"172.21.15.131:6829/475791189","heartbeat_back_addr":"172.21.15.131:6833/475791189","heartbeat_front_addr":"172.21.15.131:6831/475791189","state":["exists","up"]},{"osd":4,"uuid":"1c411bbd-9524-414b-b849-5831d601040a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":3033573590},{"type":"v1","addr":"172.21.15.150:6801","nonce":3033573590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":3033573590},{"type":"v1","addr":"172.21.15.150:6803","nonce":3033573590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":3033573590},{"type":"v1","addr":"172.21.15.150:6807","nonce":3033573590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":3033573590},{"type":"v1","addr":"172.21.15.150:6805","nonce":3033573590}]},"public_addr":"172.21.15.150:6801/3033573590","cluster_addr":"172.21.15.150:6803/3033573590","heartbeat_back_addr":"172.21.15.150:6807/3033573590","heartbeat_front_addr":"172.21.15.150:6805/3033573590","state":["exists","up"]},{"osd":5,"uuid":"a4cd7061-7b4a-46ab-89a2-e5c2c32d725e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3851715073},{"type":"v1","addr":"172.21.15.150:6809","nonce":3851715073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3851715073},{"type":"v1","addr":"172.21.15.150:6811","nonce":3851715073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3851715073},{"type":"v1","addr":"172.21.15.150:6815","nonce":3851715073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3851715073},{"type":"v1","addr":"172.21.15.150:6813","nonce":3851715073}]},"public_addr":"172.21.15.150:6809/3851715073","cluster_addr":"172.21.15.150:6811/3851715073","heartbeat_back_addr":"172.21.15.150:6815/3851715073","heartbeat_front_addr":"172.21.15.150:6813/3851715073","state":["exists","up"]},{"osd":6,"uuid":"ab5fe169-f1c9-4241-899e-fc34b5ad278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2464259658},{"type":"v1","addr":"172.21.15.150:6817","nonce":2464259658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":2464259658},{"type":"v1","addr":"172.21.15.150:6819","nonce":2464259658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2464259658},{"type":"v1","addr":"172.21.15.150:6823","nonce":2464259658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2464259658},{"type":"v1","addr":"172.21.15.150:6821","nonce":2464259658}]},"public_addr":"172.21.15.150:6817/2464259658","cluster_addr":"172.21.15.150:6819/2464259658","heartbeat_back_addr":"172.21.15.150:6823/2464259658","heartbeat_front_addr":"172.21.15.150:6821/2464259658","state":["exists","up"]},{"osd":7,"uuid":"d01488da-c749-4e79-9c3a-9e66e5a50f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4201152942},{"type":"v1","addr":"172.21.15.150:6825","nonce":4201152942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4201152942},{"type":"v1","addr":"172.21.15.150:6827","nonce":4201152942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4201152942},{"type":"v1","addr":"172.21.15.150:6831","nonce":4201152942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4201152942},{"type":"v1","addr":"172.21.15.150:6829","nonce":4201152942}]},"public_addr":"172.21.15.150:6825/4201152942","cluster_addr":"172.21.15.150:6827/4201152942","heartbeat_back_addr":"172.21.15.150:6831/4201152942","heartbeat_front_addr":"172.21.15.150:6829/4201152942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:16:30.532659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:17:00.876234+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:17:26.618705+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:17:53.047194+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:18:14.041599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:18:37.981378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:19:01.991051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:19:26.387909+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.131:6801/3558401020":"2024-02-28T06:15:25.705146+0000","172.21.15.131:6800/3558401020":"2024-02-28T06:15:25.705146+0000","172.21.15.131:0/1801222614":"2024-02-28T06:14:33.855366+0000","172.21.15.131:0/3130449061":"2024-02-28T06:15:25.705146+0000","172.21.15.131:0/2385267084":"2024-02-28T06:14:33.855366+0000","172.21.15.131:6800/4191087157":"2024-02-28T06:14:04.669941+0000","172.21.15.131:0/1817487542":"2024-02-28T06:14:33.855366+0000","172.21.15.131:0/1827492927":"2024-02-28T06:14:04.669941+0000","172.21.15.131:6801/355622241":"2024-02-28T06:14:33.855366+0000","172.21.15.131:6801/4191087157":"2024-02-28T06:14:04.669941+0000","172.21.15.131:0/1201937379":"2024-02-28T06:15:25.705146+0000","172.21.15.131:0/923129718":"2024-02-28T06:15:25.705146+0000","172.21.15.131:0/2858175649":"2024-02-28T06:14:04.669941+0000","172.21.15.131:6800/355622241":"2024-02-28T06:14:33.855366+0000","172.21.15.131:0/2215929159":"2024-02-28T06:14:04.669941+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-27T06:19:32.082 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:31 smithi131 ceph-mon[24834]: osdmap e59: 8 total, 8 up, 8 in 2024-02-27T06:19:32.083 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:31 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/529308296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T06:19:32.107 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-27T06:17:30.076310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-27T06:19:32.107 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-27T06:19:32.321 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:32.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:31 smithi150 ceph-mon[26806]: osdmap e59: 8 total, 8 up, 8 in 2024-02-27T06:19:32.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:31 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/529308296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T06:19:33.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:32 smithi131 ceph-mon[24834]: pgmap v170: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:33.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:32 smithi131 ceph-mon[24834]: osdmap e60: 8 total, 8 up, 8 in 2024-02-27T06:19:33.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:32 smithi150 ceph-mon[26806]: pgmap v170: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:33.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:32 smithi150 ceph-mon[26806]: osdmap e60: 8 total, 8 up, 8 in 2024-02-27T06:19:34.085 INFO:teuthology.orchestra.run.smithi131.stdout:pg_num: 1 2024-02-27T06:19:34.655 INFO:tasks.cephadm:Setting up client nodes... 2024-02-27T06:19:34.656 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-27T06:19:35.185 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:34 smithi131 ceph-mon[24834]: pgmap v172: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:35.185 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:34 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/3584275845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-27T06:19:35.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:34 smithi150 ceph-mon[26806]: pgmap v172: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:35.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:34 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/3584275845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-27T06:19:36.751 INFO:teuthology.orchestra.run.smithi131.stdout:[client.0] 2024-02-27T06:19:36.751 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQD4ft1lM8h1LBAAIpWat/BTGXJMSuBHXR3F/g== 2024-02-27T06:19:37.055 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:36 smithi131 ceph-mon[24834]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-27T06:19:37.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:36 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1865318156' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T06:19:37.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:36 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1865318156' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-27T06:19:37.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:36 smithi150 ceph-mon[26806]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-27T06:19:37.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:36 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1865318156' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-27T06:19:37.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:36 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1865318156' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-27T06:19:37.430 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-02-27T06:19:37.430 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-27T06:19:37.430 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-27T06:19:37.473 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-27T06:19:37.473 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-27T06:19:37.473 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph mgr dump --format=json 2024-02-27T06:19:37.724 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:39.218 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:38 smithi131 ceph-mon[24834]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-27T06:19:39.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:38 smithi150 ceph-mon[26806]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-27T06:19:39.546 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:19:40.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:39 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/348806663' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-27T06:19:40.339 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:39 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/348806663' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-27T06:19:40.365 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":19,"active_gid":14202,"active_name":"smithi131.frtgqo","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":4035631207},{"type":"v1","addr":"172.21.15.131:6801","nonce":4035631207}]},"active_addr":"172.21.15.131:6801/4035631207","active_change":"2024-02-27T06:15:25.705437+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14220,"name":"smithi150.vvgbyr","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.131:8443/","prometheus":"http://172.21.15.131:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":1904491436}]},{"addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":4083702515}]},{"addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":501434461}]},{"addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":2425988716}]}]}} 2024-02-27T06:19:40.370 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-27T06:19:40.370 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-27T06:19:40.370 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd dump --format=json 2024-02-27T06:19:40.587 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:41.017 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:40 smithi131 ceph-mon[24834]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-27T06:19:41.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:40 smithi150 ceph-mon[26806]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-27T06:19:42.222 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:19:42.223 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":60,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","created":"2024-02-27T06:13:42.247590+0000","modified":"2024-02-27T06:19:31.907958+0000","last_up_change":"2024-02-27T06:19:27.889367+0000","last_in_change":"2024-02-27T06:19:10.912604+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-27T06:17:30.076310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff509ee5-7059-4672-b377-c3eb04d10814","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":3100254707},{"type":"v1","addr":"172.21.15.131:6803","nonce":3100254707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":3100254707},{"type":"v1","addr":"172.21.15.131:6805","nonce":3100254707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3100254707},{"type":"v1","addr":"172.21.15.131:6809","nonce":3100254707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":3100254707},{"type":"v1","addr":"172.21.15.131:6807","nonce":3100254707}]},"public_addr":"172.21.15.131:6803/3100254707","cluster_addr":"172.21.15.131:6805/3100254707","heartbeat_back_addr":"172.21.15.131:6809/3100254707","heartbeat_front_addr":"172.21.15.131:6807/3100254707","state":["exists","up"]},{"osd":1,"uuid":"f3aa96ce-7431-4a6f-adca-92a510c205f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1508562642},{"type":"v1","addr":"172.21.15.131:6811","nonce":1508562642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1508562642},{"type":"v1","addr":"172.21.15.131:6813","nonce":1508562642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":1508562642},{"type":"v1","addr":"172.21.15.131:6817","nonce":1508562642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1508562642},{"type":"v1","addr":"172.21.15.131:6815","nonce":1508562642}]},"public_addr":"172.21.15.131:6811/1508562642","cluster_addr":"172.21.15.131:6813/1508562642","heartbeat_back_addr":"172.21.15.131:6817/1508562642","heartbeat_front_addr":"172.21.15.131:6815/1508562642","state":["exists","up"]},{"osd":2,"uuid":"6fb48b20-5ecd-4d7c-8784-b0fea58df271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":567909672},{"type":"v1","addr":"172.21.15.131:6819","nonce":567909672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":567909672},{"type":"v1","addr":"172.21.15.131:6821","nonce":567909672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":567909672},{"type":"v1","addr":"172.21.15.131:6825","nonce":567909672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":567909672},{"type":"v1","addr":"172.21.15.131:6823","nonce":567909672}]},"public_addr":"172.21.15.131:6819/567909672","cluster_addr":"172.21.15.131:6821/567909672","heartbeat_back_addr":"172.21.15.131:6825/567909672","heartbeat_front_addr":"172.21.15.131:6823/567909672","state":["exists","up"]},{"osd":3,"uuid":"024442d9-76aa-404a-a379-cdab69e74e65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":475791189},{"type":"v1","addr":"172.21.15.131:6827","nonce":475791189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":475791189},{"type":"v1","addr":"172.21.15.131:6829","nonce":475791189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":475791189},{"type":"v1","addr":"172.21.15.131:6833","nonce":475791189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":475791189},{"type":"v1","addr":"172.21.15.131:6831","nonce":475791189}]},"public_addr":"172.21.15.131:6827/475791189","cluster_addr":"172.21.15.131:6829/475791189","heartbeat_back_addr":"172.21.15.131:6833/475791189","heartbeat_front_addr":"172.21.15.131:6831/475791189","state":["exists","up"]},{"osd":4,"uuid":"1c411bbd-9524-414b-b849-5831d601040a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":3033573590},{"type":"v1","addr":"172.21.15.150:6801","nonce":3033573590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":3033573590},{"type":"v1","addr":"172.21.15.150:6803","nonce":3033573590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":3033573590},{"type":"v1","addr":"172.21.15.150:6807","nonce":3033573590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":3033573590},{"type":"v1","addr":"172.21.15.150:6805","nonce":3033573590}]},"public_addr":"172.21.15.150:6801/3033573590","cluster_addr":"172.21.15.150:6803/3033573590","heartbeat_back_addr":"172.21.15.150:6807/3033573590","heartbeat_front_addr":"172.21.15.150:6805/3033573590","state":["exists","up"]},{"osd":5,"uuid":"a4cd7061-7b4a-46ab-89a2-e5c2c32d725e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3851715073},{"type":"v1","addr":"172.21.15.150:6809","nonce":3851715073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3851715073},{"type":"v1","addr":"172.21.15.150:6811","nonce":3851715073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3851715073},{"type":"v1","addr":"172.21.15.150:6815","nonce":3851715073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3851715073},{"type":"v1","addr":"172.21.15.150:6813","nonce":3851715073}]},"public_addr":"172.21.15.150:6809/3851715073","cluster_addr":"172.21.15.150:6811/3851715073","heartbeat_back_addr":"172.21.15.150:6815/3851715073","heartbeat_front_addr":"172.21.15.150:6813/3851715073","state":["exists","up"]},{"osd":6,"uuid":"ab5fe169-f1c9-4241-899e-fc34b5ad278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2464259658},{"type":"v1","addr":"172.21.15.150:6817","nonce":2464259658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":2464259658},{"type":"v1","addr":"172.21.15.150:6819","nonce":2464259658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2464259658},{"type":"v1","addr":"172.21.15.150:6823","nonce":2464259658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2464259658},{"type":"v1","addr":"172.21.15.150:6821","nonce":2464259658}]},"public_addr":"172.21.15.150:6817/2464259658","cluster_addr":"172.21.15.150:6819/2464259658","heartbeat_back_addr":"172.21.15.150:6823/2464259658","heartbeat_front_addr":"172.21.15.150:6821/2464259658","state":["exists","up"]},{"osd":7,"uuid":"d01488da-c749-4e79-9c3a-9e66e5a50f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4201152942},{"type":"v1","addr":"172.21.15.150:6825","nonce":4201152942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4201152942},{"type":"v1","addr":"172.21.15.150:6827","nonce":4201152942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4201152942},{"type":"v1","addr":"172.21.15.150:6831","nonce":4201152942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4201152942},{"type":"v1","addr":"172.21.15.150:6829","nonce":4201152942}]},"public_addr":"172.21.15.150:6825/4201152942","cluster_addr":"172.21.15.150:6827/4201152942","heartbeat_back_addr":"172.21.15.150:6831/4201152942","heartbeat_front_addr":"172.21.15.150:6829/4201152942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:16:30.532659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:17:00.876234+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:17:26.618705+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:17:53.047194+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:18:14.041599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:18:37.981378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:19:01.991051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:19:26.387909+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.131:6801/3558401020":"2024-02-28T06:15:25.705146+0000","172.21.15.131:6800/3558401020":"2024-02-28T06:15:25.705146+0000","172.21.15.131:0/1801222614":"2024-02-28T06:14:33.855366+0000","172.21.15.131:0/3130449061":"2024-02-28T06:15:25.705146+0000","172.21.15.131:0/2385267084":"2024-02-28T06:14:33.855366+0000","172.21.15.131:6800/4191087157":"2024-02-28T06:14:04.669941+0000","172.21.15.131:0/1817487542":"2024-02-28T06:14:33.855366+0000","172.21.15.131:0/1827492927":"2024-02-28T06:14:04.669941+0000","172.21.15.131:6801/355622241":"2024-02-28T06:14:33.855366+0000","172.21.15.131:6801/4191087157":"2024-02-28T06:14:04.669941+0000","172.21.15.131:0/1201937379":"2024-02-28T06:15:25.705146+0000","172.21.15.131:0/923129718":"2024-02-28T06:15:25.705146+0000","172.21.15.131:0/2858175649":"2024-02-28T06:14:04.669941+0000","172.21.15.131:6800/355622241":"2024-02-28T06:14:33.855366+0000","172.21.15.131:0/2215929159":"2024-02-28T06:14:04.669941+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-27T06:19:42.920 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-27T06:19:42.920 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd dump --format=json 2024-02-27T06:19:43.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:42 smithi150 ceph-mon[26806]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-27T06:19:43.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:42 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/2737104850' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T06:19:43.133 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:43.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:42 smithi131 ceph-mon[24834]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-27T06:19:43.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:42 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2737104850' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T06:19:44.799 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:19:44.799 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":60,"fsid":"3fa4c766-d537-11ee-95c3-87774f69a715","created":"2024-02-27T06:13:42.247590+0000","modified":"2024-02-27T06:19:31.907958+0000","last_up_change":"2024-02-27T06:19:27.889367+0000","last_in_change":"2024-02-27T06:19:10.912604+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-27T06:17:30.076310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff509ee5-7059-4672-b377-c3eb04d10814","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":3100254707},{"type":"v1","addr":"172.21.15.131:6803","nonce":3100254707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":3100254707},{"type":"v1","addr":"172.21.15.131:6805","nonce":3100254707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":3100254707},{"type":"v1","addr":"172.21.15.131:6809","nonce":3100254707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":3100254707},{"type":"v1","addr":"172.21.15.131:6807","nonce":3100254707}]},"public_addr":"172.21.15.131:6803/3100254707","cluster_addr":"172.21.15.131:6805/3100254707","heartbeat_back_addr":"172.21.15.131:6809/3100254707","heartbeat_front_addr":"172.21.15.131:6807/3100254707","state":["exists","up"]},{"osd":1,"uuid":"f3aa96ce-7431-4a6f-adca-92a510c205f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":1508562642},{"type":"v1","addr":"172.21.15.131:6811","nonce":1508562642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":1508562642},{"type":"v1","addr":"172.21.15.131:6813","nonce":1508562642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":1508562642},{"type":"v1","addr":"172.21.15.131:6817","nonce":1508562642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":1508562642},{"type":"v1","addr":"172.21.15.131:6815","nonce":1508562642}]},"public_addr":"172.21.15.131:6811/1508562642","cluster_addr":"172.21.15.131:6813/1508562642","heartbeat_back_addr":"172.21.15.131:6817/1508562642","heartbeat_front_addr":"172.21.15.131:6815/1508562642","state":["exists","up"]},{"osd":2,"uuid":"6fb48b20-5ecd-4d7c-8784-b0fea58df271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":567909672},{"type":"v1","addr":"172.21.15.131:6819","nonce":567909672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":567909672},{"type":"v1","addr":"172.21.15.131:6821","nonce":567909672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":567909672},{"type":"v1","addr":"172.21.15.131:6825","nonce":567909672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":567909672},{"type":"v1","addr":"172.21.15.131:6823","nonce":567909672}]},"public_addr":"172.21.15.131:6819/567909672","cluster_addr":"172.21.15.131:6821/567909672","heartbeat_back_addr":"172.21.15.131:6825/567909672","heartbeat_front_addr":"172.21.15.131:6823/567909672","state":["exists","up"]},{"osd":3,"uuid":"024442d9-76aa-404a-a379-cdab69e74e65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":475791189},{"type":"v1","addr":"172.21.15.131:6827","nonce":475791189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":475791189},{"type":"v1","addr":"172.21.15.131:6829","nonce":475791189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":475791189},{"type":"v1","addr":"172.21.15.131:6833","nonce":475791189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":475791189},{"type":"v1","addr":"172.21.15.131:6831","nonce":475791189}]},"public_addr":"172.21.15.131:6827/475791189","cluster_addr":"172.21.15.131:6829/475791189","heartbeat_back_addr":"172.21.15.131:6833/475791189","heartbeat_front_addr":"172.21.15.131:6831/475791189","state":["exists","up"]},{"osd":4,"uuid":"1c411bbd-9524-414b-b849-5831d601040a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":3033573590},{"type":"v1","addr":"172.21.15.150:6801","nonce":3033573590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":3033573590},{"type":"v1","addr":"172.21.15.150:6803","nonce":3033573590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":3033573590},{"type":"v1","addr":"172.21.15.150:6807","nonce":3033573590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":3033573590},{"type":"v1","addr":"172.21.15.150:6805","nonce":3033573590}]},"public_addr":"172.21.15.150:6801/3033573590","cluster_addr":"172.21.15.150:6803/3033573590","heartbeat_back_addr":"172.21.15.150:6807/3033573590","heartbeat_front_addr":"172.21.15.150:6805/3033573590","state":["exists","up"]},{"osd":5,"uuid":"a4cd7061-7b4a-46ab-89a2-e5c2c32d725e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3851715073},{"type":"v1","addr":"172.21.15.150:6809","nonce":3851715073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3851715073},{"type":"v1","addr":"172.21.15.150:6811","nonce":3851715073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":3851715073},{"type":"v1","addr":"172.21.15.150:6815","nonce":3851715073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":3851715073},{"type":"v1","addr":"172.21.15.150:6813","nonce":3851715073}]},"public_addr":"172.21.15.150:6809/3851715073","cluster_addr":"172.21.15.150:6811/3851715073","heartbeat_back_addr":"172.21.15.150:6815/3851715073","heartbeat_front_addr":"172.21.15.150:6813/3851715073","state":["exists","up"]},{"osd":6,"uuid":"ab5fe169-f1c9-4241-899e-fc34b5ad278d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":2464259658},{"type":"v1","addr":"172.21.15.150:6817","nonce":2464259658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":2464259658},{"type":"v1","addr":"172.21.15.150:6819","nonce":2464259658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":2464259658},{"type":"v1","addr":"172.21.15.150:6823","nonce":2464259658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":2464259658},{"type":"v1","addr":"172.21.15.150:6821","nonce":2464259658}]},"public_addr":"172.21.15.150:6817/2464259658","cluster_addr":"172.21.15.150:6819/2464259658","heartbeat_back_addr":"172.21.15.150:6823/2464259658","heartbeat_front_addr":"172.21.15.150:6821/2464259658","state":["exists","up"]},{"osd":7,"uuid":"d01488da-c749-4e79-9c3a-9e66e5a50f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":4201152942},{"type":"v1","addr":"172.21.15.150:6825","nonce":4201152942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":4201152942},{"type":"v1","addr":"172.21.15.150:6827","nonce":4201152942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":4201152942},{"type":"v1","addr":"172.21.15.150:6831","nonce":4201152942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":4201152942},{"type":"v1","addr":"172.21.15.150:6829","nonce":4201152942}]},"public_addr":"172.21.15.150:6825/4201152942","cluster_addr":"172.21.15.150:6827/4201152942","heartbeat_back_addr":"172.21.15.150:6831/4201152942","heartbeat_front_addr":"172.21.15.150:6829/4201152942","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:16:30.532659+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:17:00.876234+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:17:26.618705+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:17:53.047194+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:18:14.041599+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:18:37.981378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:19:01.991051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-27T06:19:26.387909+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.131:6801/3558401020":"2024-02-28T06:15:25.705146+0000","172.21.15.131:6800/3558401020":"2024-02-28T06:15:25.705146+0000","172.21.15.131:0/1801222614":"2024-02-28T06:14:33.855366+0000","172.21.15.131:0/3130449061":"2024-02-28T06:15:25.705146+0000","172.21.15.131:0/2385267084":"2024-02-28T06:14:33.855366+0000","172.21.15.131:6800/4191087157":"2024-02-28T06:14:04.669941+0000","172.21.15.131:0/1817487542":"2024-02-28T06:14:33.855366+0000","172.21.15.131:0/1827492927":"2024-02-28T06:14:04.669941+0000","172.21.15.131:6801/355622241":"2024-02-28T06:14:33.855366+0000","172.21.15.131:6801/4191087157":"2024-02-28T06:14:04.669941+0000","172.21.15.131:0/1201937379":"2024-02-28T06:15:25.705146+0000","172.21.15.131:0/923129718":"2024-02-28T06:15:25.705146+0000","172.21.15.131:0/2858175649":"2024-02-28T06:14:04.669941+0000","172.21.15.131:6800/355622241":"2024-02-28T06:14:33.855366+0000","172.21.15.131:0/2215929159":"2024-02-28T06:14:04.669941+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-27T06:19:45.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:44 smithi131 ceph-mon[24834]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-27T06:19:45.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:44 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/4225481803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T06:19:45.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:44 smithi150 ceph-mon[26806]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-27T06:19:45.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:44 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/4225481803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-27T06:19:45.469 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-27T06:19:45.470 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-27T06:19:45.470 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-27T06:19:45.470 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-27T06:19:45.471 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-27T06:19:45.471 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-27T06:19:45.471 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-27T06:19:45.472 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-27T06:19:45.891 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:45.893 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:45.894 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:45.894 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:45.896 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:45.898 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:45.899 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:45.899 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:19:47.228 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:46 smithi131 ceph-mon[24834]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-27T06:19:47.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:46 smithi150 ceph-mon[26806]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-02-27T06:19:49.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:48 smithi131 ceph-mon[24834]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:49.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:48 smithi150 ceph-mon[26806]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:51.074 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:50 smithi131 ceph-mon[24834]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:51.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:50 smithi150 ceph-mon[26806]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:53.010 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:52 smithi131 ceph-mon[24834]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:53.103 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:52 smithi150 ceph-mon[26806]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:55.318 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:54 smithi131 ceph-mon[24834]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:55.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:54 smithi150 ceph-mon[26806]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:56.118 INFO:teuthology.orchestra.run.smithi131.stdout:137438953495 2024-02-27T06:19:56.119 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-27T06:19:56.566 INFO:teuthology.orchestra.run.smithi131.stdout:81604378656 2024-02-27T06:19:56.567 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-27T06:19:57.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:56 smithi131 ceph-mon[24834]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:57.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:56 smithi150 ceph-mon[26806]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:57.834 INFO:teuthology.orchestra.run.smithi131.stdout:38654705708 2024-02-27T06:19:57.834 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-27T06:19:58.573 INFO:teuthology.orchestra.run.smithi131.stdout:240518168585 2024-02-27T06:19:58.573 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-27T06:19:59.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:19:58 smithi131 ceph-mon[24834]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:59.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:19:58 smithi150 ceph-mon[26806]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:19:59.646 INFO:teuthology.orchestra.run.smithi131.stdout:206158430222 2024-02-27T06:19:59.647 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-27T06:20:00.757 INFO:teuthology.orchestra.run.smithi131.stdout:171798691859 2024-02-27T06:20:00.758 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-27T06:20:00.995 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:20:00.997 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:20:01.207 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:00 smithi131 ceph-mon[24834]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:01.207 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:00 smithi131 ceph-mon[24834]: overall HEALTH_OK 2024-02-27T06:20:01.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:00 smithi150 ceph-mon[26806]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:01.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:00 smithi150 ceph-mon[26806]: overall HEALTH_OK 2024-02-27T06:20:01.527 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:20:02.568 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:20:02.569 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:20:02.571 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:20:02.959 INFO:teuthology.orchestra.run.smithi131.stdout:60129542183 2024-02-27T06:20:02.959 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-27T06:20:03.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:02 smithi150 ceph-mon[26806]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:03.327 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:02 smithi131 ceph-mon[24834]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:05.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:04 smithi131 ceph-mon[24834]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:05.353 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:04 smithi150 ceph-mon[26806]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:06.896 INFO:teuthology.orchestra.run.smithi131.stdout:115964117021 2024-02-27T06:20:06.897 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-27T06:20:07.255 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:06 smithi131 ceph-mon[24834]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:07.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:06 smithi150 ceph-mon[26806]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:08.239 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:20:08.389 INFO:teuthology.orchestra.run.smithi131.stdout:206158430224 2024-02-27T06:20:09.186 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:08 smithi131 ceph-mon[24834]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:09.186 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:08 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2874529851' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-27T06:20:09.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:08 smithi150 ceph-mon[26806]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:09.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:08 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/2874529851' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-27T06:20:10.124 INFO:teuthology.orchestra.run.smithi131.stdout:81604378660 2024-02-27T06:20:10.331 INFO:teuthology.orchestra.run.smithi131.stdout:240518168588 2024-02-27T06:20:10.971 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168585 got 240518168588 for osd.7 2024-02-27T06:20:10.971 DEBUG:teuthology.parallel:result is None 2024-02-27T06:20:11.317 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:10 smithi131 ceph-mon[24834]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:11.317 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:10 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/4128713615' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-27T06:20:11.317 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:10 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1021871642' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-27T06:20:11.342 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378656 got 81604378660 for osd.2 2024-02-27T06:20:11.342 DEBUG:teuthology.parallel:result is None 2024-02-27T06:20:11.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:10 smithi150 ceph-mon[26806]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:11.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:10 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/4128713615' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-27T06:20:11.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:10 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1021871642' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-27T06:20:12.454 INFO:teuthology.orchestra.run.smithi131.stdout:137438953498 2024-02-27T06:20:12.639 INFO:teuthology.orchestra.run.smithi131.stdout:38654705711 2024-02-27T06:20:13.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:12 smithi150 ceph-mon[26806]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:13.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:12 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1019360340' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-27T06:20:13.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:12 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/421960847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-27T06:20:13.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:12 smithi131 ceph-mon[24834]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:13.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:12 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1019360340' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-27T06:20:13.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:12 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/421960847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-27T06:20:13.415 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:20:13.417 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705708 got 38654705711 for osd.0 2024-02-27T06:20:13.418 DEBUG:teuthology.parallel:result is None 2024-02-27T06:20:14.526 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430222 got 206158430224 for osd.6 2024-02-27T06:20:14.526 DEBUG:teuthology.parallel:result is None 2024-02-27T06:20:15.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:14 smithi131 ceph-mon[24834]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:15.109 INFO:teuthology.orchestra.run.smithi131.stdout:171798691862 2024-02-27T06:20:15.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:14 smithi150 ceph-mon[26806]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:15.768 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953495 got 137438953498 for osd.4 2024-02-27T06:20:15.769 DEBUG:teuthology.parallel:result is None 2024-02-27T06:20:16.095 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:15 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/288552385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-27T06:20:16.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:15 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/288552385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-27T06:20:16.407 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691859 got 171798691862 for osd.5 2024-02-27T06:20:16.407 DEBUG:teuthology.parallel:result is None 2024-02-27T06:20:17.048 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:16 smithi131 ceph-mon[24834]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:17.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:16 smithi150 ceph-mon[26806]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:18.752 INFO:teuthology.orchestra.run.smithi131.stdout:60129542186 2024-02-27T06:20:18.973 INFO:teuthology.orchestra.run.smithi131.stdout:115964117024 2024-02-27T06:20:19.266 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:18 smithi131 ceph-mon[24834]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:19.267 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:18 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2394268545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-27T06:20:19.267 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:18 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/3295401048' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-27T06:20:19.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:18 smithi150 ceph-mon[26806]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:19.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:18 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/2394268545' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-27T06:20:19.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:18 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/3295401048' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-27T06:20:19.540 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117021 got 115964117024 for osd.3 2024-02-27T06:20:19.540 DEBUG:teuthology.parallel:result is None 2024-02-27T06:20:20.053 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542183 got 60129542186 for osd.1 2024-02-27T06:20:20.053 DEBUG:teuthology.parallel:result is None 2024-02-27T06:20:20.053 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-27T06:20:20.053 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph pg dump --format=json 2024-02-27T06:20:20.270 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:20:21.309 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:20 smithi131 ceph-mon[24834]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:21.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:20 smithi150 ceph-mon[26806]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:22.043 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:20:22.044 INFO:teuthology.orchestra.run.smithi131.stderr:dumped all 2024-02-27T06:20:22.639 INFO:teuthology.orchestra.run.smithi131.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-02-27T06:20:21.811832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48880,"kb_used_data":6384,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":749912336,"statfs":{"total":767960285184,"available":767910232064,"internally_reserved":0,"allocated":6537216,"data_stored":3572274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002189"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":226,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-02-27T06:19:31.924229+0000","last_change":"2024-02-27T06:19:31.924229+0000","last_active":"2024-02-27T06:19:31.924229+0000","last_peered":"2024-02-27T06:19:31.924229+0000","last_clean":"2024-02-27T06:19:31.924229+0000","last_became_active":"2024-02-27T06:19:31.923757+0000","last_became_peered":"2024-02-27T06:19:31.923757+0000","last_unstale":"2024-02-27T06:19:31.924229+0000","last_undegraded":"2024-02-27T06:19:31.924229+0000","last_fullsized":"2024-02-27T06:19:31.924229+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-27T06:17:30.126433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-27T06:17:30.126433+0000","last_clean_scrub_stamp":"2024-02-27T06:17:30.126433+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-28T09:21:22.830608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6056,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739096,"statfs":{"total":95995035648,"available":95988834304,"internally_reserved":0,"allocated":1146880,"data_stored":776872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]}]},{"osd":6,"up_from":48,"seq":206158430227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":1150976,"data_stored":777195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 06:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.51300000000000001}]},{"osd":1,"last update":"Tue Feb 27 06:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52400000000000002}]},{"osd":2,"last update":"Tue Feb 27 06:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.46700000000000003}]},{"osd":3,"last update":"Tue Feb 27 06:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.5},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.44400000000000001}]},{"osd":4,"last update":"Tue Feb 27 06:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.34699999999999998}]},{"osd":5,"last update":"Tue Feb 27 06:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.40799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 06:20:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33100000000000002,"15min":0.31},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.53200000000000003,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.33100000000000002,"15min":0.30499999999999999},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.51300000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.45000000000000001}]},{"osd":2,"last update":"Tue Feb 27 06:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39200000000000002,"15min":0.38900000000000001},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.38800000000000001},"min":{"1min":0.249,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.49099999999999999}]},{"osd":3,"last update":"Tue Feb 27 06:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45700000000000002,"15min":0.46300000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.437,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.432}]},{"osd":4,"last update":"Tue Feb 27 06:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.89200000000000002}]},{"osd":5,"last update":"Tue Feb 27 06:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.875}]},{"osd":6,"last update":"Tue Feb 27 06:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1150976,"data_stored":777195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 27 06:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.33700000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53100000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.374,"15min":0.34999999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.53900000000000003}]},{"osd":2,"last update":"Tue Feb 27 06:19:33 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.39600000000000002,"15min":0.39000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.39900000000000002,"15min":0.39100000000000001},"min":{"1min":0.27600000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.48799999999999999}]},{"osd":3,"last update":"Tue Feb 27 06:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46000000000000002,"15min":0.46200000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.48099999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.51700000000000002}]},{"osd":4,"last update":"Tue Feb 27 06:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.65200000000000002}]},{"osd":5,"last update":"Tue Feb 27 06:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Tue Feb 27 06:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.621},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6244,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738908,"statfs":{"total":95995035648,"available":95988641792,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 06:19:31 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.36099999999999999,"15min":0.35799999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.39900000000000002,"15min":0.39000000000000001},"min":{"1min":0.29599999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.27800000000000002}]},{"osd":1,"last update":"Tue Feb 27 06:19:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.51500000000000001,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.377,"15min":0.36799999999999999},"min":{"1min":0.28999999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.26400000000000001}]},{"osd":3,"last update":"Tue Feb 27 06:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.44800000000000001,"15min":0.44600000000000001},"min":{"1min":0.32200000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.307,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.625,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.315}]},{"osd":4,"last update":"Tue Feb 27 06:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.55500000000000005}]},{"osd":5,"last update":"Tue Feb 27 06:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.58199999999999996}]},{"osd":6,"last update":"Tue Feb 27 06:20:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]}]},{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 06:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39900000000000002,"15min":0.40300000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.35099999999999998}]},{"osd":1,"last update":"Tue Feb 27 06:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41699999999999998,"15min":0.41899999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.61399999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.315},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.43099999999999999,"15min":0.435},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.39100000000000001}]},{"osd":2,"last update":"Tue Feb 27 06:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.30599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.433,"15min":0.434},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.627,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.29999999999999999}]},{"osd":4,"last update":"Tue Feb 27 06:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.42399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83399999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.50800000000000001}]},{"osd":5,"last update":"Tue Feb 27 06:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.499}]},{"osd":6,"last update":"Tue Feb 27 06:20:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953500,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 06:19:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.437}]},{"osd":1,"last update":"Tue Feb 27 06:19:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.497},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.44600000000000001}]},{"osd":2,"last update":"Tue Feb 27 06:19:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.56299999999999994}]},{"osd":3,"last update":"Tue Feb 27 06:19:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.46300000000000002}]},{"osd":5,"last update":"Tue Feb 27 06:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.41499999999999998}]},{"osd":6,"last update":"Tue Feb 27 06:20:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.47099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":5,"up_from":40,"seq":171798691864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5664,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739488,"statfs":{"total":95995035648,"available":95989235712,"internally_reserved":0,"allocated":614400,"data_stored":247944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 06:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.53500000000000003}]},{"osd":1,"last update":"Tue Feb 27 06:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.442},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.504}]},{"osd":2,"last update":"Tue Feb 27 06:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51700000000000002}]},{"osd":3,"last update":"Tue Feb 27 06:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41899999999999998}]},{"osd":4,"last update":"Tue Feb 27 06:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.40799999999999997}]},{"osd":6,"last update":"Tue Feb 27 06:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.52500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-27T06:20:22.642 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph pg dump --format=json 2024-02-27T06:20:22.854 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:20:23.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:22 smithi150 ceph-mon[26806]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:23.244 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:22 smithi131 ceph-mon[24834]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:24.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:23 smithi131 ceph-mon[24834]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:20:24.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:23 smithi150 ceph-mon[26806]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:20:24.657 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:20:24.658 INFO:teuthology.orchestra.run.smithi131.stderr:dumped all 2024-02-27T06:20:25.274 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:24 smithi131 ceph-mon[24834]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:25.299 INFO:teuthology.orchestra.run.smithi131.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-02-27T06:20:23.812132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48880,"kb_used_data":6384,"kb_used_omap":0,"kb_used_meta":42432,"kb_avail":749912336,"statfs":{"total":767960285184,"available":767910232064,"internally_reserved":0,"allocated":6537216,"data_stored":3572274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43450368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001907"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":226,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-02-27T06:19:31.924229+0000","last_change":"2024-02-27T06:19:31.924229+0000","last_active":"2024-02-27T06:19:31.924229+0000","last_peered":"2024-02-27T06:19:31.924229+0000","last_clean":"2024-02-27T06:19:31.924229+0000","last_became_active":"2024-02-27T06:19:31.923757+0000","last_became_peered":"2024-02-27T06:19:31.923757+0000","last_unstale":"2024-02-27T06:19:31.924229+0000","last_undegraded":"2024-02-27T06:19:31.924229+0000","last_fullsized":"2024-02-27T06:19:31.924229+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-27T06:17:30.126433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-27T06:17:30.126433+0000","last_clean_scrub_stamp":"2024-02-27T06:17:30.126433+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-28T09:21:22.830608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6056,"kb_used_data":1120,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739096,"statfs":{"total":95995035648,"available":95988834304,"internally_reserved":0,"allocated":1146880,"data_stored":776872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]}]},{"osd":6,"up_from":48,"seq":206158430227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":1150976,"data_stored":777195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 06:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.51300000000000001}]},{"osd":1,"last update":"Tue Feb 27 06:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52400000000000002}]},{"osd":2,"last update":"Tue Feb 27 06:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.46700000000000003}]},{"osd":3,"last update":"Tue Feb 27 06:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.5},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.44400000000000001}]},{"osd":4,"last update":"Tue Feb 27 06:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.34699999999999998}]},{"osd":5,"last update":"Tue Feb 27 06:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.40799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 06:20:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.33100000000000002,"15min":0.31},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.53200000000000003,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.33100000000000002,"15min":0.30499999999999999},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.51300000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.26100000000000001}]},{"osd":2,"last update":"Tue Feb 27 06:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39200000000000002,"15min":0.38900000000000001},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.38800000000000001},"min":{"1min":0.249,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.315}]},{"osd":3,"last update":"Tue Feb 27 06:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45700000000000002,"15min":0.46300000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.46100000000000002}]},{"osd":4,"last update":"Tue Feb 27 06:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.502},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.435}]},{"osd":5,"last update":"Tue Feb 27 06:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.46999999999999997}]},{"osd":6,"last update":"Tue Feb 27 06:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.52000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1150976,"data_stored":777195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 27 06:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.33700000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53100000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.374,"15min":0.34999999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.499}]},{"osd":2,"last update":"Tue Feb 27 06:19:33 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.39600000000000002,"15min":0.39000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.39900000000000002,"15min":0.39100000000000001},"min":{"1min":0.27600000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.46899999999999997}]},{"osd":3,"last update":"Tue Feb 27 06:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46000000000000002,"15min":0.46200000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.48099999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.52300000000000002}]},{"osd":4,"last update":"Tue Feb 27 06:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.88300000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.78600000000000003}]},{"osd":5,"last update":"Tue Feb 27 06:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.75900000000000001}]},{"osd":6,"last update":"Tue Feb 27 06:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.81499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6244,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738908,"statfs":{"total":95995035648,"available":95988641792,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 06:19:31 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.36099999999999999,"15min":0.35799999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.39900000000000002,"15min":0.39000000000000001},"min":{"1min":0.29599999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.27800000000000002}]},{"osd":1,"last update":"Tue Feb 27 06:19:31 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.51500000000000001,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.377,"15min":0.36799999999999999},"min":{"1min":0.28999999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.26400000000000001}]},{"osd":3,"last update":"Tue Feb 27 06:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.44800000000000001,"15min":0.44600000000000001},"min":{"1min":0.32200000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.307,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.625,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.315}]},{"osd":4,"last update":"Tue Feb 27 06:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.55500000000000005}]},{"osd":5,"last update":"Tue Feb 27 06:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.58199999999999996}]},{"osd":6,"last update":"Tue Feb 27 06:20:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]}]},{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 06:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39900000000000002,"15min":0.40300000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.35099999999999998}]},{"osd":1,"last update":"Tue Feb 27 06:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41699999999999998,"15min":0.41899999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.61399999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.315},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.43099999999999999,"15min":0.435},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.39100000000000001}]},{"osd":2,"last update":"Tue Feb 27 06:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.30599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.433,"15min":0.434},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.627,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.29999999999999999}]},{"osd":4,"last update":"Tue Feb 27 06:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.42399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83399999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.50800000000000001}]},{"osd":5,"last update":"Tue Feb 27 06:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.499}]},{"osd":6,"last update":"Tue Feb 27 06:20:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953501,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5732,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739420,"statfs":{"total":95995035648,"available":95989166080,"internally_reserved":0,"allocated":618496,"data_stored":248267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 06:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.501},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53900000000000003,"15min":0.53600000000000003},"min":{"1min":0.436,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.69999999999999996,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.69999999999999996}]},{"osd":1,"last update":"Tue Feb 27 06:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.38600000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.70299999999999996,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.64300000000000002,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.61699999999999999}]},{"osd":2,"last update":"Tue Feb 27 06:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.39800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.67400000000000004,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56499999999999995,"15min":0.56899999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.66400000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51800000000000002}]},{"osd":3,"last update":"Tue Feb 27 06:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.66200000000000003,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.67100000000000004,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.53400000000000003}]},{"osd":5,"last update":"Tue Feb 27 06:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.63}]},{"osd":6,"last update":"Tue Feb 27 06:20:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.745}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]}]},{"osd":5,"up_from":40,"seq":171798691864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5664,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739488,"statfs":{"total":95995035648,"available":95989235712,"internally_reserved":0,"allocated":614400,"data_stored":247944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 27 06:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.53500000000000003}]},{"osd":1,"last update":"Tue Feb 27 06:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.442},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.504}]},{"osd":2,"last update":"Tue Feb 27 06:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51700000000000002}]},{"osd":3,"last update":"Tue Feb 27 06:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41899999999999998}]},{"osd":4,"last update":"Tue Feb 27 06:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.40799999999999997}]},{"osd":6,"last update":"Tue Feb 27 06:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.52500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-27T06:20:25.300 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-27T06:20:25.300 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-27T06:20:25.300 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-27T06:20:25.300 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph health --format=json 2024-02-27T06:20:25.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:24 smithi150 ceph-mon[26806]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:25.516 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/mon.smithi131/config 2024-02-27T06:20:26.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:26 smithi131 ceph-mon[24834]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:20:26.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:20:26.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:20:26.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:26 smithi150 ceph-mon[26806]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:20:26.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:20:26.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:20:27.004 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:27 smithi131 ceph-mon[24834]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:27.312 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:20:27.312 INFO:teuthology.orchestra.run.smithi131.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-27T06:20:27.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:27 smithi150 ceph-mon[26806]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:27.953 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-27T06:20:27.953 INFO:tasks.cephadm:Setup complete, yielding 2024-02-27T06:20:27.954 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-27T06:20:27.964 WARNING:tasks.cephadm:rctx={'ctx': {'help': False, 'version': False, 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'block': False, 'lock': False, 'suite_path': None, 'archive': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'machine_type': 'smithi', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'owner': 'scheduled_adking@teuthology', 'os_version': None, 'verbose': True, 'os_type': None, 'interactive_on_error': False, '': True, 'config': {'archive_path': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'branch': 'wip-adk-testing-2024-02-26-1600', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'email': 'adking@redhat.com', 'first_in_suite': False, 'job_id': '7574463', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-adk-testing-2024-02-26-1600'}, 'ceph': {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-adk-testing-2024-02-26-1600', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'owner': 'scheduled_adking@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 3790, 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'sleep_before_teardown': 0, 'subset': '1/14', 'suite': 'orch:cephadm', 'suite_branch': 'wip-adk-testing-2024-02-26-1600', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'targets': {'smithi131.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyMMU2WmPTrG0nEyQ2NmvUMXtb5sj/oZhk6zkIj6rVET3wh7KaevTVGclb2I20iC0f0mf/os1Vdr5fuzKhXUrU=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1EuQQJmp6qcOuc+G8GQiW/tx9dd8YBP2iomrl9Czw/K/0oLGOOuefBVJ8PZ2KxfCk/0AouJ1k7OKERdbVOv+k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'cephadm': {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'cluster': 'ceph', 'cephadm_mode': 'root'}}, {'cephadm.shell': {'host.a': ['ceph orch status', 'ceph orch ps', 'ceph orch ls', 'ceph orch host ls', 'ceph orch device ls']}}, {'vip.exec': {'all-hosts': ['systemctl stop nfs-server']}}, {'cephadm.shell': {'host.a': ['ceph fs volume create foofs']}}, {'cephadm.wait_for_service': {'service': 'mds.foofs'}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2', 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake', 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done']}}, {'vip.exec': {'host.a': ['mkdir /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done', 'echo test > /mnt/foo/testfile', 'sync']}}, {'parallel': ['upgrade-tasks', 'workload-tasks']}, {'vip.exec': {'host.a': ['umount /mnt/foo']}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster ls | grep foo', 'ceph nfs export ls foo --detailed', 'rados -p .nfs --all ls -']}}, {'cephadm.shell': {'host.a': ['set -ex\n[[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]]\n']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-27_05:08:20', 'tube': 'smithi', 'upgrade-tasks': {'sequential': [{'cephadm.shell': {'env': ['sha1'], 'host.a': ['ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph mgr module enable nfs --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}}, {'cephadm.shell': {'env': ['sha1'], 'host.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph orch upgrade status', 'ceph health detail', 'ceph versions', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1"]}}, {'cephadm.wait_for_service': {'service': 'nfs.foo'}}]}, 'user': 'adking', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885', 'workload-tasks': {'sequential': [{'exec': {'host.a': ['cd /mnt/foo && dbench 5 -t 600 || true', 'umount /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done', 'cd /mnt/foo && dbench 5 -t 5']}}]}}, 'summary': {'success': True, 'owner': 'scheduled_adking@teuthology', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}'}, 'cluster': Cluster(remotes=[[Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.smithi131']], [Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.smithi150']]]), 'ceph': {'ceph': Namespace(admin_keyring=b'[client.admin]\n\tkey = AQCQfd1ldPHTGRAAO9qJUosu3KZYG3ESuy/Q6w==\n', bootstrap_remote=Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), bootstrapped=False, conf=ConfigObj({'global': {'log_to_file': 'true', 'log_to_stderr': 'false', 'log to journald': 'false', 'mon cluster log to file': 'true', 'mon cluster log file level': 'debug', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'debug asserts on shutdown': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'osd_pool_default_pg_autoscale_mode': 'off', 'mon allow pool delete': 'true', 'fsid': '3fa4c766-d537-11ee-95c3-87774f69a715'}, 'osd': {'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'osd memory target autotune': 'true', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug pg log writeout': 'true', 'osd debug verify cached snaps': 'true', 'osd debug verify missing on start': 'true', 'osd debug misdirected ops': 'true', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True, 'bdev debug aio': 'true', 'osd sloppy crc': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug mgr': 20, 'debug ms': 1}, 'mon': {'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'client.rgw': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true'}}), config_file='# minimal ceph.conf for 3fa4c766-d537-11ee-95c3-87774f69a715\n[global]\n\tfsid = 3fa4c766-d537-11ee-95c3-87774f69a715\n\tmon_host = [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]\n', first_mon='smithi131', first_mon_role='mon.smithi131', fsid='3fa4c766-d537-11ee-95c3-87774f69a715', image='quay.io/ceph/ceph:v17.2.0', mon_keyring=b'[mon.]\n\tkey = AQCPfd1l3PmOHxAA+kWw7vIRwPoCEi4ziUQ3cA==\n\tcaps mon = "allow *"\n', mons={'mon.smithi131': '172.21.15.131', 'mon.smithi150': '172.21.15.150'}, roleless=True, thrashers=[])}, 'cephadm': '/home/ubuntu/cephtest/cephadm', 'daemons': , 'managers': {'ceph': }, '_jinja_env': }, 'config': {'host.a': ['ceph orch status', 'ceph orch ps', 'ceph orch ls', 'ceph orch host ls', 'ceph orch device ls']}, 'cluster_name': ''} 2024-02-27T06:20:27.964 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:20:27.964 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'ceph orch status' 2024-02-27T06:20:28.268 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:28 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/3201440762' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-27T06:20:28.268 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:28 smithi131 ceph-mon[24834]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:28.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:28 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/3201440762' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-27T06:20:28.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:28 smithi150 ceph-mon[26806]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:29.859 INFO:teuthology.orchestra.run.smithi131.stdout:Backend: cephadm 2024-02-27T06:20:29.859 INFO:teuthology.orchestra.run.smithi131.stdout:Available: Yes 2024-02-27T06:20:29.859 INFO:teuthology.orchestra.run.smithi131.stdout:Paused: No 2024-02-27T06:20:30.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:20:30.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:20:30.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:29 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:20:30.185 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:20:30.186 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:20:30.186 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:29 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:20:30.496 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-27T06:20:31.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:30 smithi131 ceph-mon[24834]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:31.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:30 smithi131 ceph-mon[24834]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:20:31.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:30 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:31.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:30 smithi150 ceph-mon[26806]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:31.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:30 smithi150 ceph-mon[26806]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:20:31.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:30 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:32.497 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:20:32.497 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (4m) 2m ago 5m 15.4M - ba2b418f427c 9fbc60950b19 2024-02-27T06:20:32.497 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (5m) 2m ago 5m 6405k - 17.2.0 e1d6a67b021e 5f66626dee40 2024-02-27T06:20:32.497 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (4m) 65s ago 4m 6405k - 17.2.0 e1d6a67b021e e37d494b0a36 2024-02-27T06:20:32.497 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (4m) 2m ago 5m 41.4M - 8.3.5 dad864ee21e9 ea727583c9f8 2024-02-27T06:20:32.497 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:9283 running (6m) 2m ago 6m 449M - 17.2.0 e1d6a67b021e 8f815fe96db7 2024-02-27T06:20:32.497 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283 running (4m) 65s ago 4m 411M - 17.2.0 e1d6a67b021e c959900ba7a2 2024-02-27T06:20:32.497 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (6m) 2m ago 6m 41.2M 2048M 17.2.0 e1d6a67b021e 8e3c593dcb9f 2024-02-27T06:20:32.497 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (4m) 65s ago 4m 40.0M 2048M 17.2.0 e1d6a67b021e 976f96599f98 2024-02-27T06:20:32.498 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (5m) 2m ago 5m 21.1M - 1dbe0e931976 c20eeeeb0ea2 2024-02-27T06:20:32.498 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (4m) 65s ago 4m 16.8M - 1dbe0e931976 26b7bdf144f9 2024-02-27T06:20:32.498 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (4m) 2m ago 4m 32.1M 3216M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:20:32.498 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (3m) 2m ago 3m 34.3M 3216M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:20:32.498 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (3m) 2m ago 3m 30.9M 3216M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:20:32.498 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (2m) 2m ago 2m 10.3M 3216M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:20:32.498 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (2m) 65s ago 2m 33.7M 3984M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:20:32.498 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (119s) 65s ago 118s 33.6M 3984M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:20:32.498 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (95s) 65s ago 94s 31.1M 3984M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:20:32.498 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (70s) 65s ago 70s 13.3M 3984M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:20:32.498 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (4m) 2m ago 4m 54.3M - 514e6a882f6e 3b3a90b3ef83 2024-02-27T06:20:33.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:32 smithi131 ceph-mon[24834]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:33.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:32 smithi150 ceph-mon[26806]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:33.135 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-27T06:20:34.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:33 smithi131 ceph-mon[24834]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:20:34.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:33 smithi150 ceph-mon[26806]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:20:35.063 INFO:teuthology.orchestra.run.smithi131.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-27T06:20:35.063 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-02-27T06:20:35.063 INFO:teuthology.orchestra.run.smithi131.stdout:crash 2/2 2m ago 6m * 2024-02-27T06:20:35.063 INFO:teuthology.orchestra.run.smithi131.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-02-27T06:20:35.063 INFO:teuthology.orchestra.run.smithi131.stdout:mgr 2/2 2m ago 6m count:2 2024-02-27T06:20:35.063 INFO:teuthology.orchestra.run.smithi131.stdout:mon 2/2 2m ago 5m smithi131:172.21.15.131=smithi131;smithi150:172.21.15.150=smithi150;count:2 2024-02-27T06:20:35.064 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-02-27T06:20:35.064 INFO:teuthology.orchestra.run.smithi131.stdout:osd 8 2m ago - 2024-02-27T06:20:35.064 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-02-27T06:20:35.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:34 smithi131 ceph-mon[24834]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:35.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:34 smithi150 ceph-mon[26806]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:35.772 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-27T06:20:37.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:36 smithi131 ceph-mon[24834]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:20:37.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:36 smithi131 ceph-mon[24834]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:37.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:36 smithi150 ceph-mon[26806]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:20:37.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:36 smithi150 ceph-mon[26806]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:37.676 INFO:teuthology.orchestra.run.smithi131.stdout:HOST ADDR LABELS STATUS 2024-02-27T06:20:37.676 INFO:teuthology.orchestra.run.smithi131.stdout:smithi131 172.21.15.131 2024-02-27T06:20:37.676 INFO:teuthology.orchestra.run.smithi131.stdout:smithi150 172.21.15.150 2024-02-27T06:20:37.677 INFO:teuthology.orchestra.run.smithi131.stdout:2 hosts in cluster 2024-02-27T06:20:38.322 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-27T06:20:39.311 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:38 smithi131 ceph-mon[24834]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:20:39.311 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:38 smithi131 ceph-mon[24834]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:39.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:38 smithi150 ceph-mon[26806]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:20:39.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:38 smithi150 ceph-mon[26806]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:40.315 INFO:teuthology.orchestra.run.smithi131.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-02-27T06:20:40.315 INFO:teuthology.orchestra.run.smithi131.stdout:smithi131 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300N4400BGN 400G LVM detected, locked 2024-02-27T06:20:40.315 INFO:teuthology.orchestra.run.smithi131.stdout:smithi150 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GX400BGN 400G LVM detected, locked 2024-02-27T06:20:40.951 INFO:teuthology.run_tasks:Running task vip.exec... 2024-02-27T06:20:40.961 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:20:40.962 DEBUG:teuthology.orchestra.run.smithi131:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-02-27T06:20:40.989 INFO:teuthology.orchestra.run.smithi131.stderr:+ systemctl stop nfs-server 2024-02-27T06:20:40.997 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi150.front.sepia.ceph.com 2024-02-27T06:20:40.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-02-27T06:20:41.029 INFO:teuthology.orchestra.run.smithi150.stderr:+ systemctl stop nfs-server 2024-02-27T06:20:41.038 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-27T06:20:41.046 WARNING:tasks.cephadm:rctx={'ctx': {'help': False, 'version': False, 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'block': False, 'lock': False, 'suite_path': None, 'archive': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'machine_type': 'smithi', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'owner': 'scheduled_adking@teuthology', 'os_version': None, 'verbose': True, 'os_type': None, 'interactive_on_error': False, '': True, 'config': {'archive_path': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'branch': 'wip-adk-testing-2024-02-26-1600', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'email': 'adking@redhat.com', 'first_in_suite': False, 'job_id': '7574463', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-adk-testing-2024-02-26-1600'}, 'ceph': {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-adk-testing-2024-02-26-1600', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'owner': 'scheduled_adking@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 3790, 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'sleep_before_teardown': 0, 'subset': '1/14', 'suite': 'orch:cephadm', 'suite_branch': 'wip-adk-testing-2024-02-26-1600', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'targets': {'smithi131.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyMMU2WmPTrG0nEyQ2NmvUMXtb5sj/oZhk6zkIj6rVET3wh7KaevTVGclb2I20iC0f0mf/os1Vdr5fuzKhXUrU=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1EuQQJmp6qcOuc+G8GQiW/tx9dd8YBP2iomrl9Czw/K/0oLGOOuefBVJ8PZ2KxfCk/0AouJ1k7OKERdbVOv+k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'cephadm': {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'cluster': 'ceph', 'cephadm_mode': 'root'}}, {'cephadm.shell': {'host.a': ['ceph orch status', 'ceph orch ps', 'ceph orch ls', 'ceph orch host ls', 'ceph orch device ls']}}, {'vip.exec': {'all-hosts': ['systemctl stop nfs-server']}}, {'cephadm.shell': {'host.a': ['ceph fs volume create foofs']}}, {'cephadm.wait_for_service': {'service': 'mds.foofs'}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2', 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake', 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done']}}, {'vip.exec': {'host.a': ['mkdir /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done', 'echo test > /mnt/foo/testfile', 'sync']}}, {'parallel': ['upgrade-tasks', 'workload-tasks']}, {'vip.exec': {'host.a': ['umount /mnt/foo']}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster ls | grep foo', 'ceph nfs export ls foo --detailed', 'rados -p .nfs --all ls -']}}, {'cephadm.shell': {'host.a': ['set -ex\n[[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]]\n']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-27_05:08:20', 'tube': 'smithi', 'upgrade-tasks': {'sequential': [{'cephadm.shell': {'env': ['sha1'], 'host.a': ['ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph mgr module enable nfs --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}}, {'cephadm.shell': {'env': ['sha1'], 'host.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph orch upgrade status', 'ceph health detail', 'ceph versions', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1"]}}, {'cephadm.wait_for_service': {'service': 'nfs.foo'}}]}, 'user': 'adking', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885', 'workload-tasks': {'sequential': [{'exec': {'host.a': ['cd /mnt/foo && dbench 5 -t 600 || true', 'umount /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done', 'cd /mnt/foo && dbench 5 -t 5']}}]}}, 'summary': {'success': True, 'owner': 'scheduled_adking@teuthology', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}'}, 'cluster': Cluster(remotes=[[Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.smithi131']], [Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.smithi150']]]), 'ceph': {'ceph': Namespace(admin_keyring=b'[client.admin]\n\tkey = AQCQfd1ldPHTGRAAO9qJUosu3KZYG3ESuy/Q6w==\n', bootstrap_remote=Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), bootstrapped=False, conf=ConfigObj({'global': {'log_to_file': 'true', 'log_to_stderr': 'false', 'log to journald': 'false', 'mon cluster log to file': 'true', 'mon cluster log file level': 'debug', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'debug asserts on shutdown': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'osd_pool_default_pg_autoscale_mode': 'off', 'mon allow pool delete': 'true', 'fsid': '3fa4c766-d537-11ee-95c3-87774f69a715'}, 'osd': {'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'osd memory target autotune': 'true', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug pg log writeout': 'true', 'osd debug verify cached snaps': 'true', 'osd debug verify missing on start': 'true', 'osd debug misdirected ops': 'true', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True, 'bdev debug aio': 'true', 'osd sloppy crc': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug mgr': 20, 'debug ms': 1}, 'mon': {'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'client.rgw': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true'}}), config_file='# minimal ceph.conf for 3fa4c766-d537-11ee-95c3-87774f69a715\n[global]\n\tfsid = 3fa4c766-d537-11ee-95c3-87774f69a715\n\tmon_host = [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]\n', first_mon='smithi131', first_mon_role='mon.smithi131', fsid='3fa4c766-d537-11ee-95c3-87774f69a715', image='quay.io/ceph/ceph:v17.2.0', mon_keyring=b'[mon.]\n\tkey = AQCPfd1l3PmOHxAA+kWw7vIRwPoCEi4ziUQ3cA==\n\tcaps mon = "allow *"\n', mons={'mon.smithi131': '172.21.15.131', 'mon.smithi150': '172.21.15.150'}, roleless=True, thrashers=[])}, 'cephadm': '/home/ubuntu/cephtest/cephadm', 'daemons': , 'managers': {'ceph': }, '_jinja_env': }, 'config': {'host.a': ['ceph fs volume create foofs']}, 'cluster_name': ''} 2024-02-27T06:20:41.046 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:20:41.047 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'ceph fs volume create foofs' 2024-02-27T06:20:41.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:40 smithi131 ceph-mon[24834]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:41.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:40 smithi150 ceph-mon[26806]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:42.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:41 smithi131 ceph-mon[24834]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:20:42.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:41 smithi150 ceph-mon[26806]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:20:43.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:42 smithi150 ceph-mon[26806]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:43.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:42 smithi131 ceph-mon[24834]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:44.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:43 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-02-27T06:20:44.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:43 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-02-27T06:20:45.255 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:44 smithi150 ceph-mon[26806]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:20:45.256 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:44 smithi150 ceph-mon[26806]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:45.256 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:44 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-02-27T06:20:45.256 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:44 smithi150 ceph-mon[26806]: osdmap e61: 8 total, 8 up, 8 in 2024-02-27T06:20:45.256 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:44 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-02-27T06:20:45.323 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:44 smithi131 ceph-mon[24834]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:20:45.323 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:44 smithi131 ceph-mon[24834]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:45.323 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:44 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-02-27T06:20:45.323 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:44 smithi131 ceph-mon[24834]: osdmap e61: 8 total, 8 up, 8 in 2024-02-27T06:20:45.323 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:44 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-02-27T06:20:45.323 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:44 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131[24830]: 2024-02-27T06:20:44.947+0000 7f4bd4541700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-27T06:20:45.646 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-02-27T06:20:45.656 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-02-27T06:20:45.656 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch ls -f json 2024-02-27T06:20:46.219 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-02-27T06:20:46.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: osdmap e62: 8 total, 8 up, 8 in 2024-02-27T06:20:46.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-02-27T06:20:46.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-27T06:20:46.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-27T06:20:46.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: osdmap e63: 8 total, 8 up, 8 in 2024-02-27T06:20:46.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-02-27T06:20:46.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: fsmap foofs:0 2024-02-27T06:20:46.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: Saving service mds.foofs spec with placement count:2 2024-02-27T06:20:46.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:46.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:20:46.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:20:46.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:20:46.221 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:46.221 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:46.221 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi131.mhuwwb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-27T06:20:46.221 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi131.mhuwwb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-27T06:20:46.221 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:45 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:20:46.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-02-27T06:20:46.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: osdmap e62: 8 total, 8 up, 8 in 2024-02-27T06:20:46.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-02-27T06:20:46.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-27T06:20:46.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-27T06:20:46.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: osdmap e63: 8 total, 8 up, 8 in 2024-02-27T06:20:46.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-02-27T06:20:46.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: fsmap foofs:0 2024-02-27T06:20:46.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: Saving service mds.foofs spec with placement count:2 2024-02-27T06:20:46.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:46.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:20:46.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:20:46.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:20:46.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:46.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:46.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi131.mhuwwb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-27T06:20:46.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi131.mhuwwb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-27T06:20:46.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:45 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:20:47.286 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:46 smithi131 ceph-mon[24834]: Deploying daemon mds.foofs.smithi131.mhuwwb on smithi131 2024-02-27T06:20:47.286 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:46 smithi131 ceph-mon[24834]: pgmap v211: 65 pgs: 5 creating+peering, 1 active+clean, 59 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:47.286 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:46 smithi131 ceph-mon[24834]: osdmap e64: 8 total, 8 up, 8 in 2024-02-27T06:20:47.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:46 smithi150 ceph-mon[26806]: Deploying daemon mds.foofs.smithi131.mhuwwb on smithi131 2024-02-27T06:20:47.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:46 smithi150 ceph-mon[26806]: pgmap v211: 65 pgs: 5 creating+peering, 1 active+clean, 59 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:47.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:46 smithi150 ceph-mon[26806]: osdmap e64: 8 total, 8 up, 8 in 2024-02-27T06:20:48.131 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:47 smithi131 ceph-mon[24834]: osdmap e65: 8 total, 8 up, 8 in 2024-02-27T06:20:48.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:47 smithi150 ceph-mon[26806]: osdmap e65: 8 total, 8 up, 8 in 2024-02-27T06:20:48.820 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:20:48.820 INFO:teuthology.orchestra.run.smithi131.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-27T06:14:23.661249Z", "last_refresh": "2024-02-27T06:17:55.580760Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-27T06:14:17.375793Z", "last_refresh": "2024-02-27T06:17:55.580824Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-02-27T06:14:20.285100Z", "last_refresh": "2024-02-27T06:17:55.580886Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-02-27T06:20:44.965043Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-27T06:20:44.960815Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-27T06:14:15.518198Z", "last_refresh": "2024-02-27T06:17:55.580690Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi131:172.21.15.131=smithi131", "smithi150:172.21.15.150=smithi150"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-27T06:15:04.040710Z", "last_refresh": "2024-02-27T06:17:55.580543Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-27T06:14:21.429969Z", "last_refresh": "2024-02-27T06:17:55.580947Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-02-27T06:17:55.581081Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-27T06:14:18.623960Z", "last_refresh": "2024-02-27T06:17:55.581017Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-27T06:20:49.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:48 smithi131 ceph-mon[24834]: pgmap v214: 65 pgs: 6 creating+activating, 5 creating+peering, 27 active+clean, 27 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:49.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:48 smithi150 ceph-mon[26806]: pgmap v214: 65 pgs: 6 creating+activating, 5 creating+peering, 27 active+clean, 27 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:49.499 INFO:tasks.cephadm:mds.foofs has 0/2 2024-02-27T06:20:50.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:49 smithi150 ceph-mon[26806]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:20:50.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:49 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:50.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:49 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi150.vsnjhu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-27T06:20:50.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:49 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi150.vsnjhu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-27T06:20:50.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:49 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:20:50.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:49 smithi131 ceph-mon[24834]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:20:50.350 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:49 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:50.350 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:49 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi150.vsnjhu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-27T06:20:50.350 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:49 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi150.vsnjhu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-27T06:20:50.350 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:49 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:20:50.500 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch ls -f json 2024-02-27T06:20:51.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:50 smithi131 ceph-mon[24834]: Deploying daemon mds.foofs.smithi150.vsnjhu on smithi150 2024-02-27T06:20:51.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:50 smithi131 ceph-mon[24834]: pgmap v215: 65 pgs: 6 creating+activating, 5 creating+peering, 52 active+clean, 2 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:51.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:50 smithi131 ceph-mon[24834]: mds.? [v2:172.21.15.131:6834/3029043434,v1:172.21.15.131:6835/3029043434] up:boot 2024-02-27T06:20:51.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:50 smithi131 ceph-mon[24834]: daemon mds.foofs.smithi131.mhuwwb assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-02-27T06:20:51.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:50 smithi131 ceph-mon[24834]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-27T06:20:51.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:50 smithi131 ceph-mon[24834]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-27T06:20:51.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:50 smithi131 ceph-mon[24834]: Cluster is now healthy 2024-02-27T06:20:51.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:50 smithi131 ceph-mon[24834]: fsmap foofs:0 1 up:standby 2024-02-27T06:20:51.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:50 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:20:51.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:50 smithi131 ceph-mon[24834]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:creating} 2024-02-27T06:20:51.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:50 smithi131 ceph-mon[24834]: daemon mds.foofs.smithi131.mhuwwb is now active in filesystem foofs as rank 0 2024-02-27T06:20:51.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:50 smithi150 ceph-mon[26806]: Deploying daemon mds.foofs.smithi150.vsnjhu on smithi150 2024-02-27T06:20:51.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:50 smithi150 ceph-mon[26806]: pgmap v215: 65 pgs: 6 creating+activating, 5 creating+peering, 52 active+clean, 2 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:51.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:50 smithi150 ceph-mon[26806]: mds.? [v2:172.21.15.131:6834/3029043434,v1:172.21.15.131:6835/3029043434] up:boot 2024-02-27T06:20:51.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:50 smithi150 ceph-mon[26806]: daemon mds.foofs.smithi131.mhuwwb assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-02-27T06:20:51.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:50 smithi150 ceph-mon[26806]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-27T06:20:51.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:50 smithi150 ceph-mon[26806]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-27T06:20:51.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:50 smithi150 ceph-mon[26806]: Cluster is now healthy 2024-02-27T06:20:51.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:50 smithi150 ceph-mon[26806]: fsmap foofs:0 1 up:standby 2024-02-27T06:20:51.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:50 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:20:51.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:50 smithi150 ceph-mon[26806]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:creating} 2024-02-27T06:20:51.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:50 smithi150 ceph-mon[26806]: daemon mds.foofs.smithi131.mhuwwb is now active in filesystem foofs as rank 0 2024-02-27T06:20:52.099 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:51 smithi150 ceph-mon[26806]: mds.? [v2:172.21.15.131:6834/3029043434,v1:172.21.15.131:6835/3029043434] up:active 2024-02-27T06:20:52.099 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:51 smithi150 ceph-mon[26806]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 2024-02-27T06:20:52.246 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:51 smithi131 ceph-mon[24834]: mds.? [v2:172.21.15.131:6834/3029043434,v1:172.21.15.131:6835/3029043434] up:active 2024-02-27T06:20:52.246 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:51 smithi131 ceph-mon[24834]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 2024-02-27T06:20:52.685 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:20:52.685 INFO:teuthology.orchestra.run.smithi131.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-27T06:14:23.661249Z", "last_refresh": "2024-02-27T06:17:55.580760Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-27T06:14:17.375793Z", "last_refresh": "2024-02-27T06:17:55.580824Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-02-27T06:14:20.285100Z", "last_refresh": "2024-02-27T06:17:55.580886Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-02-27T06:20:44.965043Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-27T06:20:44.960815Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-27T06:14:15.518198Z", "last_refresh": "2024-02-27T06:17:55.580690Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi131:172.21.15.131=smithi131", "smithi150:172.21.15.150=smithi150"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-27T06:15:04.040710Z", "last_refresh": "2024-02-27T06:17:55.580543Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-27T06:14:21.429969Z", "last_refresh": "2024-02-27T06:17:55.580947Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-02-27T06:17:55.581081Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-27T06:14:18.623960Z", "last_refresh": "2024-02-27T06:17:55.581017Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-27T06:20:53.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:52 smithi150 ceph-mon[26806]: pgmap v216: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:53.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:52 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:53.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:52 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:20:53.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:52 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:20:53.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:52 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:20:53.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:52 smithi131 ceph-mon[24834]: pgmap v216: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-02-27T06:20:53.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:52 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:53.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:52 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:20:53.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:52 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:20:53.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:52 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:20:53.975 INFO:tasks.cephadm:mds.foofs has 0/2 2024-02-27T06:20:54.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:53 smithi131 ceph-mon[24834]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:20:54.259 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:53 smithi131 ceph-mon[24834]: mds.? [v2:172.21.15.150:6832/3371494173,v1:172.21.15.150:6833/3371494173] up:boot 2024-02-27T06:20:54.259 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:53 smithi131 ceph-mon[24834]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 1 up:standby 2024-02-27T06:20:54.259 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:53 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:20:54.259 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:53 smithi131 ceph-mon[24834]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 1 up:standby 2024-02-27T06:20:54.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:53 smithi150 ceph-mon[26806]: from='client.14484 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:20:54.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:53 smithi150 ceph-mon[26806]: mds.? [v2:172.21.15.150:6832/3371494173,v1:172.21.15.150:6833/3371494173] up:boot 2024-02-27T06:20:54.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:53 smithi150 ceph-mon[26806]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 1 up:standby 2024-02-27T06:20:54.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:53 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:20:54.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:53 smithi150 ceph-mon[26806]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 1 up:standby 2024-02-27T06:20:54.975 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch ls -f json 2024-02-27T06:20:55.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:54 smithi131 ceph-mon[24834]: pgmap v217: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-02-27T06:20:55.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:54 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:55.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:54 smithi150 ceph-mon[26806]: pgmap v217: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-02-27T06:20:55.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:54 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:57.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:56 smithi131 ceph-mon[24834]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-27T06:20:57.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:56 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:57.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:56 smithi150 ceph-mon[26806]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-27T06:20:57.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:56 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:20:57.836 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:20:57.837 INFO:teuthology.orchestra.run.smithi131.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-27T06:14:23.661249Z", "last_refresh": "2024-02-27T06:17:55.580760Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-27T06:14:17.375793Z", "last_refresh": "2024-02-27T06:17:55.580824Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-02-27T06:14:20.285100Z", "last_refresh": "2024-02-27T06:17:55.580886Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-02-27T06:20:44.965043Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-27T06:20:44.960815Z", "last_refresh": "2024-02-27T06:20:54.117834Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-27T06:14:15.518198Z", "last_refresh": "2024-02-27T06:17:55.580690Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi131:172.21.15.131=smithi131", "smithi150:172.21.15.150=smithi150"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-27T06:15:04.040710Z", "last_refresh": "2024-02-27T06:17:55.580543Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-27T06:14:21.429969Z", "last_refresh": "2024-02-27T06:17:55.580947Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-02-27T06:17:55.581081Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-27T06:14:18.623960Z", "last_refresh": "2024-02-27T06:17:55.581017Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-27T06:20:58.638 INFO:tasks.cephadm:mds.foofs has 1/2 2024-02-27T06:20:59.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:58 smithi131 ceph-mon[24834]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-02-27T06:20:59.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:20:58 smithi131 ceph-mon[24834]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:20:59.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:58 smithi150 ceph-mon[26806]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-02-27T06:20:59.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:20:58 smithi150 ceph-mon[26806]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:20:59.639 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch ls -f json 2024-02-27T06:21:00.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:00.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:00.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:21:00.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:00.245 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:21:00.246 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:00 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:00.246 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:00 smithi131 ceph-mon[24834]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-27T06:21:00.604 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:00.605 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:00.605 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:21:00.605 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:00.605 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:21:00.605 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:00 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:00.605 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:00 smithi150 ceph-mon[26806]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-27T06:21:01.740 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:21:01.740 INFO:teuthology.orchestra.run.smithi131.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-27T06:14:23.661249Z", "last_refresh": "2024-02-27T06:20:59.134055Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-27T06:14:17.375793Z", "last_refresh": "2024-02-27T06:20:54.117231Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-02-27T06:14:20.285100Z", "last_refresh": "2024-02-27T06:20:59.134183Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-02-27T06:20:44.965043Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-27T06:20:44.960815Z", "last_refresh": "2024-02-27T06:20:54.117834Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-27T06:14:15.518198Z", "last_refresh": "2024-02-27T06:20:54.117392Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi131:172.21.15.131=smithi131", "smithi150:172.21.15.150=smithi150"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-27T06:15:04.040710Z", "last_refresh": "2024-02-27T06:20:54.117464Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-27T06:14:21.429969Z", "last_refresh": "2024-02-27T06:20:54.117529Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-02-27T06:20:54.117591Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-27T06:14:18.623960Z", "last_refresh": "2024-02-27T06:20:59.134320Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-27T06:21:02.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:01 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:02.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:01 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:02.431 INFO:tasks.cephadm:mds.foofs has 2/2 2024-02-27T06:21:02.431 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-27T06:21:02.441 WARNING:tasks.cephadm:rctx={'ctx': {'help': False, 'version': False, 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'block': False, 'lock': False, 'suite_path': None, 'archive': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'machine_type': 'smithi', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'owner': 'scheduled_adking@teuthology', 'os_version': None, 'verbose': True, 'os_type': None, 'interactive_on_error': False, '': True, 'config': {'archive_path': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'branch': 'wip-adk-testing-2024-02-26-1600', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'email': 'adking@redhat.com', 'first_in_suite': False, 'job_id': '7574463', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-adk-testing-2024-02-26-1600'}, 'ceph': {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-adk-testing-2024-02-26-1600', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'owner': 'scheduled_adking@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 3790, 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'sleep_before_teardown': 0, 'subset': '1/14', 'suite': 'orch:cephadm', 'suite_branch': 'wip-adk-testing-2024-02-26-1600', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'targets': {'smithi131.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyMMU2WmPTrG0nEyQ2NmvUMXtb5sj/oZhk6zkIj6rVET3wh7KaevTVGclb2I20iC0f0mf/os1Vdr5fuzKhXUrU=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1EuQQJmp6qcOuc+G8GQiW/tx9dd8YBP2iomrl9Czw/K/0oLGOOuefBVJ8PZ2KxfCk/0AouJ1k7OKERdbVOv+k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'cephadm': {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'cluster': 'ceph', 'cephadm_mode': 'root'}}, {'cephadm.shell': {'host.a': ['ceph orch status', 'ceph orch ps', 'ceph orch ls', 'ceph orch host ls', 'ceph orch device ls']}}, {'vip.exec': {'all-hosts': ['systemctl stop nfs-server']}}, {'cephadm.shell': {'host.a': ['ceph fs volume create foofs']}}, {'cephadm.wait_for_service': {'service': 'mds.foofs'}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2', 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake', 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done']}}, {'vip.exec': {'host.a': ['mkdir /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done', 'echo test > /mnt/foo/testfile', 'sync']}}, {'parallel': ['upgrade-tasks', 'workload-tasks']}, {'vip.exec': {'host.a': ['umount /mnt/foo']}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster ls | grep foo', 'ceph nfs export ls foo --detailed', 'rados -p .nfs --all ls -']}}, {'cephadm.shell': {'host.a': ['set -ex\n[[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]]\n']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-27_05:08:20', 'tube': 'smithi', 'upgrade-tasks': {'sequential': [{'cephadm.shell': {'env': ['sha1'], 'host.a': ['ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph mgr module enable nfs --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}}, {'cephadm.shell': {'env': ['sha1'], 'host.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph orch upgrade status', 'ceph health detail', 'ceph versions', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1"]}}, {'cephadm.wait_for_service': {'service': 'nfs.foo'}}]}, 'user': 'adking', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885', 'workload-tasks': {'sequential': [{'exec': {'host.a': ['cd /mnt/foo && dbench 5 -t 600 || true', 'umount /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done', 'cd /mnt/foo && dbench 5 -t 5']}}]}}, 'summary': {'success': True, 'owner': 'scheduled_adking@teuthology', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}'}, 'cluster': Cluster(remotes=[[Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.smithi131']], [Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.smithi150']]]), 'ceph': {'ceph': Namespace(admin_keyring=b'[client.admin]\n\tkey = AQCQfd1ldPHTGRAAO9qJUosu3KZYG3ESuy/Q6w==\n', bootstrap_remote=Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), bootstrapped=False, conf=ConfigObj({'global': {'log_to_file': 'true', 'log_to_stderr': 'false', 'log to journald': 'false', 'mon cluster log to file': 'true', 'mon cluster log file level': 'debug', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'debug asserts on shutdown': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'osd_pool_default_pg_autoscale_mode': 'off', 'mon allow pool delete': 'true', 'fsid': '3fa4c766-d537-11ee-95c3-87774f69a715'}, 'osd': {'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'osd memory target autotune': 'true', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug pg log writeout': 'true', 'osd debug verify cached snaps': 'true', 'osd debug verify missing on start': 'true', 'osd debug misdirected ops': 'true', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True, 'bdev debug aio': 'true', 'osd sloppy crc': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug mgr': 20, 'debug ms': 1}, 'mon': {'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'client.rgw': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true'}}), config_file='# minimal ceph.conf for 3fa4c766-d537-11ee-95c3-87774f69a715\n[global]\n\tfsid = 3fa4c766-d537-11ee-95c3-87774f69a715\n\tmon_host = [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]\n', first_mon='smithi131', first_mon_role='mon.smithi131', fsid='3fa4c766-d537-11ee-95c3-87774f69a715', image='quay.io/ceph/ceph:v17.2.0', mon_keyring=b'[mon.]\n\tkey = AQCPfd1l3PmOHxAA+kWw7vIRwPoCEi4ziUQ3cA==\n\tcaps mon = "allow *"\n', mons={'mon.smithi131': '172.21.15.131', 'mon.smithi150': '172.21.15.150'}, roleless=True, thrashers=[])}, 'cephadm': '/home/ubuntu/cephtest/cephadm', 'daemons': , 'managers': {'ceph': }, '_jinja_env': }, 'config': {'host.a': ['ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2', 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake', 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done']}, 'cluster_name': ''} 2024-02-27T06:21:02.441 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:21:02.442 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-02-27T06:21:02.976 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:02 smithi131 ceph-mon[24834]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:21:02.977 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:02 smithi131 ceph-mon[24834]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-27T06:21:03.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:02 smithi150 ceph-mon[26806]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:21:03.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:02 smithi150 ceph-mon[26806]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-27T06:21:05.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:04 smithi131 ceph-mon[24834]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-27T06:21:05.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:04 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-02-27T06:21:05.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:04 smithi150 ceph-mon[26806]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-27T06:21:05.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:04 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-02-27T06:21:05.939 INFO:teuthology.orchestra.run.smithi131.stdout:NFS Cluster Created Successfully 2024-02-27T06:21:06.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:05 smithi131 ceph-mon[24834]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:06.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:05 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-02-27T06:21:06.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:05 smithi131 ceph-mon[24834]: osdmap e66: 8 total, 8 up, 8 in 2024-02-27T06:21:06.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:05 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-02-27T06:21:06.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:05 smithi150 ceph-mon[26806]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:06.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:05 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-02-27T06:21:06.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:05 smithi150 ceph-mon[26806]: osdmap e66: 8 total, 8 up, 8 in 2024-02-27T06:21:06.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:05 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-02-27T06:21:06.680 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-02-27T06:21:07.209 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: pgmap v224: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-02-27T06:21:07.209 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-02-27T06:21:07.209 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: osdmap e67: 8 total, 8 up, 8 in 2024-02-27T06:21:07.209 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: Saving service nfs.foo spec with placement count:2 2024-02-27T06:21:07.209 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:07.209 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:21:07.209 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:07.209 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:21:07.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:07.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:07.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi131.cezbzx", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-27T06:21:07.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi131.cezbzx", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-02-27T06:21:07.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-27T06:21:07.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-27T06:21:07.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:07.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-27T06:21:07.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-27T06:21:07.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi131.cezbzx-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T06:21:07.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi131.cezbzx-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T06:21:07.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:06 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:07.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: pgmap v224: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-02-27T06:21:07.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-02-27T06:21:07.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: osdmap e67: 8 total, 8 up, 8 in 2024-02-27T06:21:07.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: Saving service nfs.foo spec with placement count:2 2024-02-27T06:21:07.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:07.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:21:07.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:07.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:21:07.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:07.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:07.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi131.cezbzx", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-27T06:21:07.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi131.cezbzx", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-02-27T06:21:07.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-27T06:21:07.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-27T06:21:07.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:07.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-27T06:21:07.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-27T06:21:07.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi131.cezbzx-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T06:21:07.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi131.cezbzx-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T06:21:07.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:06 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:08.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:07 smithi131 ceph-mon[24834]: Creating key for client.nfs.foo.0.0.smithi131.cezbzx 2024-02-27T06:21:08.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:07 smithi131 ceph-mon[24834]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-02-27T06:21:08.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:07 smithi131 ceph-mon[24834]: Rados config object exists: conf-nfs.foo 2024-02-27T06:21:08.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:07 smithi131 ceph-mon[24834]: Creating key for client.nfs.foo.0.0.smithi131.cezbzx-rgw 2024-02-27T06:21:08.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:07 smithi131 ceph-mon[24834]: Deploying daemon nfs.foo.0.0.smithi131.cezbzx on smithi131 2024-02-27T06:21:08.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:07 smithi131 ceph-mon[24834]: osdmap e68: 8 total, 8 up, 8 in 2024-02-27T06:21:08.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:07 smithi150 ceph-mon[26806]: Creating key for client.nfs.foo.0.0.smithi131.cezbzx 2024-02-27T06:21:08.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:07 smithi150 ceph-mon[26806]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-02-27T06:21:08.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:07 smithi150 ceph-mon[26806]: Rados config object exists: conf-nfs.foo 2024-02-27T06:21:08.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:07 smithi150 ceph-mon[26806]: Creating key for client.nfs.foo.0.0.smithi131.cezbzx-rgw 2024-02-27T06:21:08.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:07 smithi150 ceph-mon[26806]: Deploying daemon nfs.foo.0.0.smithi131.cezbzx on smithi131 2024-02-27T06:21:08.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:07 smithi150 ceph-mon[26806]: osdmap e68: 8 total, 8 up, 8 in 2024-02-27T06:21:09.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:08 smithi131 ceph-mon[24834]: pgmap v227: 97 pgs: 6 creating+activating, 9 unknown, 82 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-02-27T06:21:09.220 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:08 smithi131 ceph-mon[24834]: mgrmap e20: smithi131.frtgqo(active, since 5m), standbys: smithi150.vvgbyr 2024-02-27T06:21:09.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:08 smithi150 ceph-mon[26806]: pgmap v227: 97 pgs: 6 creating+activating, 9 unknown, 82 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-02-27T06:21:09.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:08 smithi150 ceph-mon[26806]: mgrmap e20: smithi131.frtgqo(active, since 5m), standbys: smithi150.vvgbyr 2024-02-27T06:21:09.740 INFO:teuthology.orchestra.run.smithi131.stderr:Invalid command: Unexpected argument '--clusterid' 2024-02-27T06:21:09.740 INFO:teuthology.orchestra.run.smithi131.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-02-27T06:21:09.740 INFO:teuthology.orchestra.run.smithi131.stderr:Error EINVAL: invalid command 2024-02-27T06:21:10.121 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:21:10.121 INFO:teuthology.orchestra.run.smithi131.stdout: "bind": "/fake", 2024-02-27T06:21:10.121 INFO:teuthology.orchestra.run.smithi131.stdout: "fs": "foofs", 2024-02-27T06:21:10.121 INFO:teuthology.orchestra.run.smithi131.stdout: "path": "/", 2024-02-27T06:21:10.122 INFO:teuthology.orchestra.run.smithi131.stdout: "cluster": "foo", 2024-02-27T06:21:10.122 INFO:teuthology.orchestra.run.smithi131.stdout: "mode": "RW" 2024-02-27T06:21:10.122 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:21:10.704 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-02-27T06:21:11.034 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:10 smithi131 ceph-mon[24834]: pgmap v228: 97 pgs: 6 creating+activating, 3 unknown, 88 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-02-27T06:21:11.034 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:11.035 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi150.xgkkez", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-27T06:21:11.035 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi150.xgkkez", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-02-27T06:21:11.035 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-27T06:21:11.035 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-27T06:21:11.035 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:11.035 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-02-27T06:21:11.035 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-02-27T06:21:11.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:10 smithi150 ceph-mon[26806]: pgmap v228: 97 pgs: 6 creating+activating, 3 unknown, 88 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-02-27T06:21:11.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:11.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi150.xgkkez", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-27T06:21:11.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi150.xgkkez", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-02-27T06:21:11.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-27T06:21:11.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-27T06:21:11.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:11.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-02-27T06:21:11.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-02-27T06:21:12.214 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:11 smithi131 ceph-mon[24834]: Creating key for client.nfs.foo.1.0.smithi150.xgkkez 2024-02-27T06:21:12.214 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:11 smithi131 ceph-mon[24834]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-02-27T06:21:12.214 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:11 smithi131 ceph-mon[24834]: from='client.14518 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:12.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:11 smithi150 ceph-mon[26806]: Creating key for client.nfs.foo.1.0.smithi150.xgkkez 2024-02-27T06:21:12.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:11 smithi150 ceph-mon[26806]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-02-27T06:21:12.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:11 smithi150 ceph-mon[26806]: from='client.14518 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:13.073 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:12 smithi150 ceph-mon[26806]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-02-27T06:21:13.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:12 smithi131 ceph-mon[24834]: pgmap v229: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-02-27T06:21:14.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:13 smithi150 ceph-mon[26806]: from='client.14534 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:14.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:13 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-27T06:21:14.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:13 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-27T06:21:14.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:13 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi150.xgkkez-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T06:21:14.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:13 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi150.xgkkez-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T06:21:14.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:13 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:14.183 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:13 smithi131 ceph-mon[24834]: from='client.14534 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:14.183 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:13 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-27T06:21:14.183 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:13 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-27T06:21:14.183 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:13 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi150.xgkkez-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T06:21:14.183 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:13 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi150.xgkkez-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-27T06:21:14.183 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:13 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:15.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:14 smithi150 ceph-mon[26806]: Rados config object exists: conf-nfs.foo 2024-02-27T06:21:15.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:14 smithi150 ceph-mon[26806]: Creating key for client.nfs.foo.1.0.smithi150.xgkkez-rgw 2024-02-27T06:21:15.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:14 smithi150 ceph-mon[26806]: Deploying daemon nfs.foo.1.0.smithi150.xgkkez on smithi150 2024-02-27T06:21:15.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:14 smithi150 ceph-mon[26806]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-27T06:21:15.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:14 smithi131 ceph-mon[24834]: Rados config object exists: conf-nfs.foo 2024-02-27T06:21:15.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:14 smithi131 ceph-mon[24834]: Creating key for client.nfs.foo.1.0.smithi150.xgkkez-rgw 2024-02-27T06:21:15.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:14 smithi131 ceph-mon[24834]: Deploying daemon nfs.foo.1.0.smithi150.xgkkez on smithi150 2024-02-27T06:21:15.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:14 smithi131 ceph-mon[24834]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-27T06:21:16.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:15 smithi131 ceph-mon[24834]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:16.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:15 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:16.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:15 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:21:16.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:15 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:16.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:15 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:21:16.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:15 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:16.239 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:15 smithi150 ceph-mon[26806]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:16.239 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:15 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:16.239 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:15 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:21:16.239 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:15 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:16.239 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:15 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:21:16.240 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:15 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:17.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:16 smithi131 ceph-mon[24834]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:17.349 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:16 smithi131 ceph-mon[24834]: pgmap v231: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-27T06:21:17.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:16 smithi150 ceph-mon[26806]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:17.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:16 smithi150 ceph-mon[26806]: pgmap v231: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-27T06:21:18.019 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:17 smithi150 ceph-mon[26806]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:18.337 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:17 smithi131 ceph-mon[24834]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:19.242 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:18 smithi131 ceph-mon[24834]: pgmap v232: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-02-27T06:21:19.243 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:18 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:19.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:18 smithi150 ceph-mon[26806]: pgmap v232: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-02-27T06:21:19.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:18 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:20.080 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:19 smithi131 ceph-mon[24834]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:20.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:19 smithi150 ceph-mon[26806]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:20.732 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo ?:2049 2/2 2s ago 14s count:2 2024-02-27T06:21:21.046 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:20 smithi131 ceph-mon[24834]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:21.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:20 smithi131 ceph-mon[24834]: pgmap v233: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-02-27T06:21:21.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:20 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:21.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:20 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:21.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:20 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:21.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:20 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:21:21.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:20 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:21.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:20 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:21:21.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:20 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:21.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:20 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:21.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:20 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:21.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:20 smithi150 ceph-mon[26806]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:21.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:20 smithi150 ceph-mon[26806]: pgmap v233: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-02-27T06:21:21.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:20 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:21.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:20 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:21.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:20 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:21.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:20 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:21:21.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:20 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:21.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:20 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:21:21.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:20 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:21.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:20 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:21.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:20 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:21.399 INFO:teuthology.run_tasks:Running task vip.exec... 2024-02-27T06:21:21.410 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:21:21.410 DEBUG:teuthology.orchestra.run.smithi131:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-02-27T06:21:21.438 INFO:teuthology.orchestra.run.smithi131.stderr:+ mkdir /mnt/foo 2024-02-27T06:21:21.451 DEBUG:teuthology.orchestra.run.smithi131:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-02-27T06:21:21.518 INFO:teuthology.orchestra.run.smithi131.stderr:++ hostname 2024-02-27T06:21:21.518 INFO:teuthology.orchestra.run.smithi131.stderr:+ mount -t nfs smithi131:/fake /mnt/foo -o sync 2024-02-27T06:21:21.919 DEBUG:teuthology.orchestra.run.smithi131:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-02-27T06:21:21.987 INFO:teuthology.orchestra.run.smithi131.stderr:+ echo test 2024-02-27T06:21:22.030 DEBUG:teuthology.orchestra.run.smithi131:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-02-27T06:21:22.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:21 smithi131 ceph-mon[24834]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:22.096 INFO:teuthology.orchestra.run.smithi131.stderr:+ sync 2024-02-27T06:21:22.252 INFO:teuthology.run_tasks:Running task parallel... 2024-02-27T06:21:22.263 INFO:teuthology.task.parallel:starting parallel... 2024-02-27T06:21:22.263 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-27T06:21:22.264 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-27T06:21:22.264 WARNING:tasks.cephadm:rctx={'ctx': {'help': False, 'version': False, 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'block': False, 'lock': False, 'suite_path': None, 'archive': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'machine_type': 'smithi', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'owner': 'scheduled_adking@teuthology', 'os_version': None, 'verbose': True, 'os_type': None, 'interactive_on_error': False, '': True, 'config': {'archive_path': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'branch': 'wip-adk-testing-2024-02-26-1600', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'email': 'adking@redhat.com', 'first_in_suite': False, 'job_id': '7574463', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-adk-testing-2024-02-26-1600'}, 'ceph': {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-adk-testing-2024-02-26-1600', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'owner': 'scheduled_adking@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 3790, 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'sleep_before_teardown': 0, 'subset': '1/14', 'suite': 'orch:cephadm', 'suite_branch': 'wip-adk-testing-2024-02-26-1600', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'targets': {'smithi131.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyMMU2WmPTrG0nEyQ2NmvUMXtb5sj/oZhk6zkIj6rVET3wh7KaevTVGclb2I20iC0f0mf/os1Vdr5fuzKhXUrU=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1EuQQJmp6qcOuc+G8GQiW/tx9dd8YBP2iomrl9Czw/K/0oLGOOuefBVJ8PZ2KxfCk/0AouJ1k7OKERdbVOv+k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'cephadm': {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'cluster': 'ceph', 'cephadm_mode': 'root'}}, {'cephadm.shell': {'host.a': ['ceph orch status', 'ceph orch ps', 'ceph orch ls', 'ceph orch host ls', 'ceph orch device ls']}}, {'vip.exec': {'all-hosts': ['systemctl stop nfs-server']}}, {'cephadm.shell': {'host.a': ['ceph fs volume create foofs']}}, {'cephadm.wait_for_service': {'service': 'mds.foofs'}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2', 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake', 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done']}}, {'vip.exec': {'host.a': ['mkdir /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done', 'echo test > /mnt/foo/testfile', 'sync']}}, {'parallel': ['upgrade-tasks', 'workload-tasks']}, {'vip.exec': {'host.a': ['umount /mnt/foo']}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster ls | grep foo', 'ceph nfs export ls foo --detailed', 'rados -p .nfs --all ls -']}}, {'cephadm.shell': {'host.a': ['set -ex\n[[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]]\n']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-27_05:08:20', 'tube': 'smithi', 'upgrade-tasks': {'sequential': [{'cephadm.shell': {'host.a': ['ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph mgr module enable nfs --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}}, {'cephadm.shell': {'env': ['sha1'], 'host.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph orch upgrade status', 'ceph health detail', 'ceph versions', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1"]}}, {'cephadm.wait_for_service': {'service': 'nfs.foo'}}]}, 'user': 'adking', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885', 'workload-tasks': {'sequential': [{'exec': {'host.a': ['cd /mnt/foo && dbench 5 -t 600 || true', 'umount /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done', 'cd /mnt/foo && dbench 5 -t 5']}}]}}, 'summary': {'success': True, 'owner': 'scheduled_adking@teuthology', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}'}, 'cluster': Cluster(remotes=[[Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.smithi131']], [Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.smithi150']]]), 'ceph': {'ceph': Namespace(admin_keyring=b'[client.admin]\n\tkey = AQCQfd1ldPHTGRAAO9qJUosu3KZYG3ESuy/Q6w==\n', bootstrap_remote=Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), bootstrapped=False, conf=ConfigObj({'global': {'log_to_file': 'true', 'log_to_stderr': 'false', 'log to journald': 'false', 'mon cluster log to file': 'true', 'mon cluster log file level': 'debug', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'debug asserts on shutdown': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'osd_pool_default_pg_autoscale_mode': 'off', 'mon allow pool delete': 'true', 'fsid': '3fa4c766-d537-11ee-95c3-87774f69a715'}, 'osd': {'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'osd memory target autotune': 'true', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug pg log writeout': 'true', 'osd debug verify cached snaps': 'true', 'osd debug verify missing on start': 'true', 'osd debug misdirected ops': 'true', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True, 'bdev debug aio': 'true', 'osd sloppy crc': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug mgr': 20, 'debug ms': 1}, 'mon': {'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'client.rgw': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true'}}), config_file='# minimal ceph.conf for 3fa4c766-d537-11ee-95c3-87774f69a715\n[global]\n\tfsid = 3fa4c766-d537-11ee-95c3-87774f69a715\n\tmon_host = [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]\n', first_mon='smithi131', first_mon_role='mon.smithi131', fsid='3fa4c766-d537-11ee-95c3-87774f69a715', image='quay.io/ceph/ceph:v17.2.0', mon_keyring=b'[mon.]\n\tkey = AQCPfd1l3PmOHxAA+kWw7vIRwPoCEi4ziUQ3cA==\n\tcaps mon = "allow *"\n', mons={'mon.smithi131': '172.21.15.131', 'mon.smithi150': '172.21.15.150'}, roleless=True, thrashers=[])}, 'cephadm': '/home/ubuntu/cephtest/cephadm', 'daemons': , 'managers': {'ceph': }, '_jinja_env': }, 'config': {'host.a': ['ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph mgr module enable nfs --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}, 'cluster_name': ''} 2024-02-27T06:21:22.265 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:21:22.265 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-27T06:21:22.265 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-27T06:21:22.265 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-02-27T06:21:22.266 INFO:teuthology.task.exec:Executing custom commands... 2024-02-27T06:21:22.266 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:21:22.266 DEBUG:teuthology.orchestra.run.smithi131:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-02-27T06:21:22.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:21 smithi150 ceph-mon[26806]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:22.386 INFO:teuthology.orchestra.run.smithi131.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-02-27T06:21:22.386 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:21:22.386 INFO:teuthology.orchestra.run.smithi131.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-02-27T06:21:22.387 INFO:teuthology.orchestra.run.smithi131.stdout:3 of 5 processes prepared for launch 0 sec 2024-02-27T06:21:22.487 INFO:teuthology.orchestra.run.smithi131.stdout:5 of 5 processes prepared for launch 0 sec 2024-02-27T06:21:22.488 INFO:teuthology.orchestra.run.smithi131.stdout:releasing clients 2024-02-27T06:21:23.099 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:22 smithi131 ceph-mon[24834]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-02-27T06:21:23.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:22 smithi150 ceph-mon[26806]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-02-27T06:21:23.490 INFO:teuthology.orchestra.run.smithi131.stdout: 5 130 28.81 MB/sec warmup 1 sec latency 31.483 ms 2024-02-27T06:21:24.490 INFO:teuthology.orchestra.run.smithi131.stdout: 5 280 26.14 MB/sec warmup 2 sec latency 32.131 ms 2024-02-27T06:21:25.185 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:24 smithi131 ceph-mon[24834]: pgmap v235: 97 pgs: 97 active+clean; 3.8 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 280 KiB/s wr, 9 op/s 2024-02-27T06:21:25.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:24 smithi150 ceph-mon[26806]: pgmap v235: 97 pgs: 97 active+clean; 3.8 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 280 KiB/s wr, 9 op/s 2024-02-27T06:21:25.490 INFO:teuthology.orchestra.run.smithi131.stdout: 5 430 25.19 MB/sec warmup 3 sec latency 32.657 ms 2024-02-27T06:21:25.634 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-27T06:21:26.232 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-02-27T06:21:26.232 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-27T06:21:26.232 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-27T06:21:26.232 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-27T06:21:26.232 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:21:26.232 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:25 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:21:26.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-02-27T06:21:26.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-27T06:21:26.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-27T06:21:26.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-27T06:21:26.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:21:26.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:25 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:21:26.491 INFO:teuthology.orchestra.run.smithi131.stdout: 5 573 24.47 MB/sec warmup 4 sec latency 35.655 ms 2024-02-27T06:21:27.009 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:26 smithi131 ceph-mon[24834]: pgmap v236: 97 pgs: 97 active+clean; 18 MiB data, 63 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.6 MiB/s wr, 136 op/s 2024-02-27T06:21:27.009 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-02-27T06:21:27.009 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-27T06:21:27.009 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-27T06:21:27.009 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-02-27T06:21:27.009 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:26 smithi131 ceph-mon[24834]: osdmap e69: 8 total, 8 up, 8 in 2024-02-27T06:21:27.225 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:26 smithi150 ceph-mon[26806]: pgmap v236: 97 pgs: 97 active+clean; 18 MiB data, 63 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.6 MiB/s wr, 136 op/s 2024-02-27T06:21:27.225 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-02-27T06:21:27.225 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-27T06:21:27.225 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-27T06:21:27.225 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-02-27T06:21:27.225 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:26 smithi150 ceph-mon[26806]: osdmap e69: 8 total, 8 up, 8 in 2024-02-27T06:21:27.491 INFO:teuthology.orchestra.run.smithi131.stdout: 5 959 21.89 MB/sec warmup 5 sec latency 28.538 ms 2024-02-27T06:21:28.020 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:27 smithi131 ceph-mon[24834]: osdmap e70: 8 total, 8 up, 8 in 2024-02-27T06:21:28.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:27 smithi150 ceph-mon[26806]: osdmap e70: 8 total, 8 up, 8 in 2024-02-27T06:21:28.491 INFO:teuthology.orchestra.run.smithi131.stdout: 5 2092 22.20 MB/sec warmup 6 sec latency 12.736 ms 2024-02-27T06:21:29.024 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:28 smithi131 ceph-mon[24834]: pgmap v239: 97 pgs: 1 activating+degraded, 96 active+clean; 79 MiB data, 114 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 10 MiB/s wr, 406 op/s; 1/723 objects degraded (0.138%) 2024-02-27T06:21:29.024 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:28 smithi131 ceph-mon[24834]: Health check failed: Degraded data redundancy: 1/723 objects degraded (0.138%), 1 pg degraded (PG_DEGRADED) 2024-02-27T06:21:29.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:28 smithi150 ceph-mon[26806]: pgmap v239: 97 pgs: 1 activating+degraded, 96 active+clean; 79 MiB data, 114 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 10 MiB/s wr, 406 op/s; 1/723 objects degraded (0.138%) 2024-02-27T06:21:29.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:28 smithi150 ceph-mon[26806]: Health check failed: Degraded data redundancy: 1/723 objects degraded (0.138%), 1 pg degraded (PG_DEGRADED) 2024-02-27T06:21:29.491 INFO:teuthology.orchestra.run.smithi131.stdout: 5 3188 24.59 MB/sec warmup 7 sec latency 10.919 ms 2024-02-27T06:21:30.491 INFO:teuthology.orchestra.run.smithi131.stdout: 5 3984 25.03 MB/sec warmup 8 sec latency 14.037 ms 2024-02-27T06:21:30.600 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-27T06:21:31.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:30 smithi131 ceph-mon[24834]: pgmap v240: 97 pgs: 1 activating+degraded, 96 active+clean; 107 MiB data, 146 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 14 MiB/s wr, 600 op/s; 1/1086 objects degraded (0.092%) 2024-02-27T06:21:31.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:30 smithi150 ceph-mon[26806]: pgmap v240: 97 pgs: 1 activating+degraded, 96 active+clean; 107 MiB data, 146 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 14 MiB/s wr, 600 op/s; 1/1086 objects degraded (0.092%) 2024-02-27T06:21:31.491 INFO:teuthology.orchestra.run.smithi131.stdout: 5 4921 24.41 MB/sec warmup 9 sec latency 13.079 ms 2024-02-27T06:21:32.491 INFO:teuthology.orchestra.run.smithi131.stdout: 5 5974 23.67 MB/sec warmup 10 sec latency 11.385 ms 2024-02-27T06:21:33.044 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:32 smithi131 ceph-mon[24834]: pgmap v241: 97 pgs: 1 activating+degraded, 96 active+clean; 120 MiB data, 186 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 16 MiB/s wr, 828 op/s; 1/1389 objects degraded (0.072%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-27T06:21:33.104 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:32 smithi150 ceph-mon[26806]: pgmap v241: 97 pgs: 1 activating+degraded, 96 active+clean; 120 MiB data, 186 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 16 MiB/s wr, 828 op/s; 1/1389 objects degraded (0.072%); 0 B/s, 0 keys/s, 0 objects/s recovering 2024-02-27T06:21:33.491 INFO:teuthology.orchestra.run.smithi131.stdout: 5 6877 24.84 MB/sec warmup 11 sec latency 72.207 ms 2024-02-27T06:21:34.080 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:34 smithi131 ceph-mon[24834]: pgmap v242: 97 pgs: 97 active+clean; 131 MiB data, 243 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 18 MiB/s wr, 882 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-02-27T06:21:34.080 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:34 smithi131 ceph-mon[24834]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1389 objects degraded (0.072%), 1 pg degraded) 2024-02-27T06:21:34.080 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:34 smithi131 ceph-mon[24834]: Cluster is now healthy 2024-02-27T06:21:34.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:34 smithi150 ceph-mon[26806]: pgmap v242: 97 pgs: 97 active+clean; 131 MiB data, 243 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 18 MiB/s wr, 882 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-02-27T06:21:34.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:34 smithi150 ceph-mon[26806]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1389 objects degraded (0.072%), 1 pg degraded) 2024-02-27T06:21:34.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:34 smithi150 ceph-mon[26806]: Cluster is now healthy 2024-02-27T06:21:34.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 7570 24.66 MB/sec warmup 12 sec latency 44.572 ms 2024-02-27T06:21:35.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 8498 24.27 MB/sec warmup 13 sec latency 13.522 ms 2024-02-27T06:21:35.788 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph mgr module enable nfs --force' 2024-02-27T06:21:36.491 INFO:teuthology.orchestra.run.smithi131.stdout: 5 9483 23.62 MB/sec warmup 14 sec latency 19.466 ms 2024-02-27T06:21:37.112 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:36 smithi131 ceph-mon[24834]: pgmap v243: 97 pgs: 97 active+clean; 138 MiB data, 300 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 16 MiB/s wr, 942 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-02-27T06:21:37.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:36 smithi150 ceph-mon[26806]: pgmap v243: 97 pgs: 97 active+clean; 138 MiB data, 300 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 16 MiB/s wr, 942 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-02-27T06:21:37.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 10560 25.15 MB/sec warmup 15 sec latency 15.632 ms 2024-02-27T06:21:38.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 11246 24.50 MB/sec warmup 16 sec latency 29.982 ms 2024-02-27T06:21:39.121 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:38 smithi131 ceph-mon[24834]: pgmap v244: 97 pgs: 97 active+clean; 142 MiB data, 362 MiB used, 715 GiB / 715 GiB avail; 6.6 KiB/s rd, 16 MiB/s wr, 1.01k op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-02-27T06:21:39.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:38 smithi150 ceph-mon[26806]: pgmap v244: 97 pgs: 97 active+clean; 142 MiB data, 362 MiB used, 715 GiB / 715 GiB avail; 6.6 KiB/s rd, 16 MiB/s wr, 1.01k op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-02-27T06:21:39.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 12070 24.19 MB/sec warmup 17 sec latency 11.415 ms 2024-02-27T06:21:39.921 INFO:teuthology.orchestra.run.smithi131.stderr:module 'nfs' is already enabled 2024-02-27T06:21:40.125 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:39 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1049486594' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-02-27T06:21:40.125 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:39 smithi131 ceph-mon[24834]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-02-27T06:21:40.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:39 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1049486594' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-02-27T06:21:40.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:39 smithi150 ceph-mon[26806]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-02-27T06:21:40.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 13076 23.78 MB/sec warmup 18 sec latency 17.651 ms 2024-02-27T06:21:41.133 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:40 smithi131 ceph-mon[24834]: pgmap v245: 97 pgs: 97 active+clean; 153 MiB data, 410 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 14 MiB/s wr, 1.14k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-27T06:21:41.133 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:40 smithi131 ceph-mon[24834]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-02-27T06:21:41.133 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:40 smithi131 ceph-mon[24834]: mgrmap e21: smithi131.frtgqo(active, since 6m), standbys: smithi150.vvgbyr 2024-02-27T06:21:41.276 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-27T06:21:41.354 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:40 smithi150 ceph-mon[26806]: pgmap v245: 97 pgs: 97 active+clean; 153 MiB data, 410 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 14 MiB/s wr, 1.14k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-27T06:21:41.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:40 smithi150 ceph-mon[26806]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-02-27T06:21:41.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:40 smithi150 ceph-mon[26806]: mgrmap e21: smithi131.frtgqo(active, since 6m), standbys: smithi150.vvgbyr 2024-02-27T06:21:41.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 14097 24.86 MB/sec warmup 19 sec latency 18.769 ms 2024-02-27T06:21:42.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 14699 24.36 MB/sec warmup 20 sec latency 18.487 ms 2024-02-27T06:21:43.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:42 smithi150 ceph-mon[26806]: pgmap v246: 97 pgs: 97 active+clean; 161 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.18k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-27T06:21:43.146 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:42 smithi131 ceph-mon[24834]: pgmap v246: 97 pgs: 97 active+clean; 161 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.18k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-02-27T06:21:43.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 15315 23.60 MB/sec warmup 21 sec latency 94.508 ms 2024-02-27T06:21:44.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 16455 23.61 MB/sec warmup 22 sec latency 62.814 ms 2024-02-27T06:21:45.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:44 smithi131 ceph-mon[24834]: pgmap v247: 97 pgs: 97 active+clean; 166 MiB data, 480 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 14 MiB/s wr, 1.21k op/s; 1 B/s, 0 objects/s recovering 2024-02-27T06:21:45.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:44 smithi150 ceph-mon[26806]: pgmap v247: 97 pgs: 97 active+clean; 166 MiB data, 480 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 14 MiB/s wr, 1.21k op/s; 1 B/s, 0 objects/s recovering 2024-02-27T06:21:45.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 17279 24.02 MB/sec warmup 23 sec latency 17.966 ms 2024-02-27T06:21:45.834 INFO:teuthology.orchestra.run.smithi131.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T06:21:46.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 17855 24.06 MB/sec warmup 24 sec latency 31.030 ms 2024-02-27T06:21:46.948 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:46 smithi131 ceph-mon[24834]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:46.948 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:46 smithi131 ceph-mon[24834]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T06:21:46.948 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:46 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:46.949 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:46 smithi131 ceph-mon[24834]: pgmap v248: 97 pgs: 97 active+clean; 169 MiB data, 506 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 1.17k op/s 2024-02-27T06:21:46.949 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:46 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:21:46.949 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:46 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:46.949 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:46 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:21:46.949 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:46 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:46.949 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:46 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:47.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:46 smithi150 ceph-mon[26806]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:47.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:46 smithi150 ceph-mon[26806]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T06:21:47.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:46 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:47.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:46 smithi150 ceph-mon[26806]: pgmap v248: 97 pgs: 97 active+clean; 169 MiB data, 506 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 1.17k op/s 2024-02-27T06:21:47.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:46 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:21:47.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:46 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:21:47.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:46 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:21:47.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:46 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:47.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:46 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:21:47.220 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-27T06:21:47.221 WARNING:tasks.cephadm:rctx={'ctx': {'help': False, 'version': False, 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'block': False, 'lock': False, 'suite_path': None, 'archive': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'machine_type': 'smithi', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'owner': 'scheduled_adking@teuthology', 'os_version': None, 'verbose': True, 'os_type': None, 'interactive_on_error': False, '': True, 'config': {'archive_path': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'branch': 'wip-adk-testing-2024-02-26-1600', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'email': 'adking@redhat.com', 'first_in_suite': False, 'job_id': '7574463', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-adk-testing-2024-02-26-1600'}, 'ceph': {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-adk-testing-2024-02-26-1600', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'owner': 'scheduled_adking@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 3790, 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'sleep_before_teardown': 0, 'subset': '1/14', 'suite': 'orch:cephadm', 'suite_branch': 'wip-adk-testing-2024-02-26-1600', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'targets': {'smithi131.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyMMU2WmPTrG0nEyQ2NmvUMXtb5sj/oZhk6zkIj6rVET3wh7KaevTVGclb2I20iC0f0mf/os1Vdr5fuzKhXUrU=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1EuQQJmp6qcOuc+G8GQiW/tx9dd8YBP2iomrl9Czw/K/0oLGOOuefBVJ8PZ2KxfCk/0AouJ1k7OKERdbVOv+k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'cephadm': {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'cluster': 'ceph', 'cephadm_mode': 'root'}}, {'cephadm.shell': {'host.a': ['ceph orch status', 'ceph orch ps', 'ceph orch ls', 'ceph orch host ls', 'ceph orch device ls']}}, {'vip.exec': {'all-hosts': ['systemctl stop nfs-server']}}, {'cephadm.shell': {'host.a': ['ceph fs volume create foofs']}}, {'cephadm.wait_for_service': {'service': 'mds.foofs'}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2', 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake', 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done']}}, {'vip.exec': {'host.a': ['mkdir /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done', 'echo test > /mnt/foo/testfile', 'sync']}}, {'parallel': ['upgrade-tasks', 'workload-tasks']}, {'vip.exec': {'host.a': ['umount /mnt/foo']}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster ls | grep foo', 'ceph nfs export ls foo --detailed', 'rados -p .nfs --all ls -']}}, {'cephadm.shell': {'host.a': ['set -ex\n[[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]]\n']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-27_05:08:20', 'tube': 'smithi', 'upgrade-tasks': {'sequential': [{'cephadm.shell': {'host.a': ['ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph mgr module enable nfs --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}}, {'cephadm.shell': {'host.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph orch upgrade status', 'ceph health detail', 'ceph versions', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1"]}}, {'cephadm.wait_for_service': {'service': 'nfs.foo'}}]}, 'user': 'adking', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885', 'workload-tasks': {'sequential': [{'exec': {'host.a': ['cd /mnt/foo && dbench 5 -t 600 || true', 'umount /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done', 'cd /mnt/foo && dbench 5 -t 5']}}]}}, 'summary': {'success': True, 'owner': 'scheduled_adking@teuthology', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}'}, 'cluster': Cluster(remotes=[[Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.smithi131']], [Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.smithi150']]]), 'ceph': {'ceph': Namespace(admin_keyring=b'[client.admin]\n\tkey = AQCQfd1ldPHTGRAAO9qJUosu3KZYG3ESuy/Q6w==\n', bootstrap_remote=Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), bootstrapped=False, conf=ConfigObj({'global': {'log_to_file': 'true', 'log_to_stderr': 'false', 'log to journald': 'false', 'mon cluster log to file': 'true', 'mon cluster log file level': 'debug', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'debug asserts on shutdown': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'osd_pool_default_pg_autoscale_mode': 'off', 'mon allow pool delete': 'true', 'fsid': '3fa4c766-d537-11ee-95c3-87774f69a715'}, 'osd': {'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'osd memory target autotune': 'true', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug pg log writeout': 'true', 'osd debug verify cached snaps': 'true', 'osd debug verify missing on start': 'true', 'osd debug misdirected ops': 'true', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True, 'bdev debug aio': 'true', 'osd sloppy crc': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug mgr': 20, 'debug ms': 1}, 'mon': {'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'client.rgw': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true'}}), config_file='# minimal ceph.conf for 3fa4c766-d537-11ee-95c3-87774f69a715\n[global]\n\tfsid = 3fa4c766-d537-11ee-95c3-87774f69a715\n\tmon_host = [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]\n', first_mon='smithi131', first_mon_role='mon.smithi131', fsid='3fa4c766-d537-11ee-95c3-87774f69a715', image='quay.io/ceph/ceph:v17.2.0', mon_keyring=b'[mon.]\n\tkey = AQCPfd1l3PmOHxAA+kWw7vIRwPoCEi4ziUQ3cA==\n\tcaps mon = "allow *"\n', mons={'mon.smithi131': '172.21.15.131', 'mon.smithi150': '172.21.15.150'}, roleless=True, thrashers=[])}, 'cephadm': '/home/ubuntu/cephtest/cephadm', 'daemons': , 'managers': {'ceph': }, '_jinja_env': }, 'config': {'host.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph orch upgrade status', 'ceph health detail', 'ceph versions', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1"]}, 'cluster_name': ''} 2024-02-27T06:21:47.221 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:21:47.221 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-27T06:21:47.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 18140 23.28 MB/sec warmup 25 sec latency 34.670 ms 2024-02-27T06:21:47.959 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:47 smithi131 ceph-mon[24834]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T06:21:48.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:47 smithi150 ceph-mon[26806]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 2024-02-27T06:21:48.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 18647 22.61 MB/sec warmup 26 sec latency 34.194 ms 2024-02-27T06:21:48.970 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:48 smithi131 ceph-mon[24834]: pgmap v249: 97 pgs: 97 active+clean; 173 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 14 MiB/s wr, 1.23k op/s 2024-02-27T06:21:49.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:48 smithi150 ceph-mon[26806]: pgmap v249: 97 pgs: 97 active+clean; 173 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 14 MiB/s wr, 1.23k op/s 2024-02-27T06:21:49.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 19192 22.32 MB/sec warmup 27 sec latency 54.117 ms 2024-02-27T06:21:50.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 19942 21.92 MB/sec warmup 28 sec latency 58.640 ms 2024-02-27T06:21:50.997 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:50 smithi131 ceph-mon[24834]: pgmap v250: 97 pgs: 97 active+clean; 176 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 1.15k op/s 2024-02-27T06:21:51.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:50 smithi150 ceph-mon[26806]: pgmap v250: 97 pgs: 97 active+clean; 176 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 12 MiB/s wr, 1.15k op/s 2024-02-27T06:21:51.367 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:21:51.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 20429 21.69 MB/sec warmup 29 sec latency 32.155 ms 2024-02-27T06:21:52.368 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:21:52.368 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (5m) 31s ago 6m 22.8M - ba2b418f427c 9fbc60950b19 2024-02-27T06:21:52.368 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (6m) 31s ago 6m 6405k - 17.2.0 e1d6a67b021e 5f66626dee40 2024-02-27T06:21:52.368 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (6m) 34s ago 6m 6405k - 17.2.0 e1d6a67b021e e37d494b0a36 2024-02-27T06:21:52.368 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (5m) 31s ago 6m 50.2M - 8.3.5 dad864ee21e9 ea727583c9f8 2024-02-27T06:21:52.368 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (63s) 31s ago 62s 13.5M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:21:52.368 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (60s) 34s ago 60s 11.8M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:21:52.368 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:9283 running (8m) 31s ago 8m 462M - 17.2.0 e1d6a67b021e 8f815fe96db7 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283 running (6m) 34s ago 6m 412M - 17.2.0 e1d6a67b021e c959900ba7a2 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (8m) 31s ago 8m 46.4M 2048M 17.2.0 e1d6a67b021e 8e3c593dcb9f 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (6m) 34s ago 6m 37.5M 2048M 17.2.0 e1d6a67b021e 976f96599f98 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (42s) 31s ago 42s 51.3M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (37s) 34s ago 36s 48.8M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (6m) 31s ago 6m 17.2M - 1dbe0e931976 c20eeeeb0ea2 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (6m) 34s ago 6m 16.5M - 1dbe0e931976 26b7bdf144f9 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (5m) 31s ago 5m 42.2M 3216M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (4m) 31s ago 4m 43.0M 3216M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (4m) 31s ago 4m 40.7M 3216M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (4m) 31s ago 4m 39.2M 3216M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (3m) 34s ago 3m 40.1M 3984M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (3m) 34s ago 3m 39.7M 3984M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:21:52.369 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (2m) 34s ago 2m 39.5M 3984M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:21:52.370 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (2m) 34s ago 2m 43.1M 3984M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:21:52.370 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (5m) 31s ago 5m 65.2M - 514e6a882f6e 3b3a90b3ef83 2024-02-27T06:21:52.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 21151 21.96 MB/sec warmup 30 sec latency 23.021 ms 2024-02-27T06:21:52.998 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:21:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:21:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:21:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:21:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:21:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:21:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:21:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:21:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T06:21:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:21:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:21:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:21:52.999 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:21:52.999 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:21:52.999 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-02-27T06:21:52.999 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:21:52.999 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:21:53.074 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:52 smithi131 ceph-mon[24834]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:53.074 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:52 smithi131 ceph-mon[24834]: pgmap v251: 97 pgs: 97 active+clean; 176 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 936 op/s 2024-02-27T06:21:53.074 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:52 smithi131 ceph-mon[24834]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:53.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:52 smithi150 ceph-mon[26806]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:53.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:52 smithi150 ceph-mon[26806]: pgmap v251: 97 pgs: 97 active+clean; 176 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 936 op/s 2024-02-27T06:21:53.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:52 smithi150 ceph-mon[26806]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:53.478 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:21:53.478 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32", 2024-02-27T06:21:53.478 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:21:53.478 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [], 2024-02-27T06:21:53.478 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "", 2024-02-27T06:21:53.478 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:635ee6850c57c98cc6ba932cade45f9cc015aa32 image" 2024-02-27T06:21:53.479 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:21:53.492 INFO:teuthology.orchestra.run.smithi131.stdout: 5 21726 21.87 MB/sec warmup 31 sec latency 21.190 ms 2024-02-27T06:21:54.080 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:53 smithi131 ceph-mon[24834]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:54.080 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:53 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/53004479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:21:54.134 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:21:54.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:53 smithi150 ceph-mon[26806]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:54.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:53 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/53004479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:21:54.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 22176 21.32 MB/sec warmup 32 sec latency 37.510 ms 2024-02-27T06:21:55.088 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:54 smithi131 ceph-mon[24834]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:55.088 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:54 smithi131 ceph-mon[24834]: pgmap v252: 97 pgs: 97 active+clean; 178 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 905 op/s 2024-02-27T06:21:55.088 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:54 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1679173131' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:21:55.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:54 smithi150 ceph-mon[26806]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:21:55.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:54 smithi150 ceph-mon[26806]: pgmap v252: 97 pgs: 97 active+clean; 178 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 905 op/s 2024-02-27T06:21:55.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:54 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1679173131' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:21:55.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 22616 21.03 MB/sec warmup 33 sec latency 25.430 ms 2024-02-27T06:21:56.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 23237 20.79 MB/sec warmup 34 sec latency 36.975 ms 2024-02-27T06:21:57.102 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:56 smithi131 ceph-mon[24834]: pgmap v253: 97 pgs: 97 active+clean; 179 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 8.2 MiB/s wr, 757 op/s 2024-02-27T06:21:57.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:56 smithi150 ceph-mon[26806]: pgmap v253: 97 pgs: 97 active+clean; 179 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 8.2 MiB/s wr, 757 op/s 2024-02-27T06:21:57.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 23858 20.55 MB/sec warmup 35 sec latency 26.723 ms 2024-02-27T06:21:58.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 24408 20.66 MB/sec warmup 36 sec latency 45.179 ms 2024-02-27T06:21:59.145 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:21:58 smithi131 ceph-mon[24834]: pgmap v254: 97 pgs: 97 active+clean; 187 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 841 op/s 2024-02-27T06:21:59.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:21:58 smithi150 ceph-mon[26806]: pgmap v254: 97 pgs: 97 active+clean; 187 MiB data, 675 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 841 op/s 2024-02-27T06:21:59.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 24999 20.71 MB/sec warmup 37 sec latency 25.993 ms 2024-02-27T06:22:00.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 25362 20.38 MB/sec warmup 38 sec latency 31.579 ms 2024-02-27T06:22:01.182 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:00 smithi131 ceph-mon[24834]: pgmap v255: 97 pgs: 97 active+clean; 195 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 8.1 MiB/s wr, 817 op/s 2024-02-27T06:22:01.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:00 smithi150 ceph-mon[26806]: pgmap v255: 97 pgs: 97 active+clean; 195 MiB data, 736 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 8.1 MiB/s wr, 817 op/s 2024-02-27T06:22:01.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 25849 20.00 MB/sec warmup 39 sec latency 31.612 ms 2024-02-27T06:22:02.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 26630 19.98 MB/sec warmup 40 sec latency 39.671 ms 2024-02-27T06:22:03.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:02 smithi150 ceph-mon[26806]: pgmap v256: 97 pgs: 97 active+clean; 193 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 720 op/s 2024-02-27T06:22:03.191 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:02 smithi131 ceph-mon[24834]: pgmap v256: 97 pgs: 97 active+clean; 193 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 720 op/s 2024-02-27T06:22:03.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 27404 19.94 MB/sec warmup 41 sec latency 24.617 ms 2024-02-27T06:22:04.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 28244 20.25 MB/sec warmup 42 sec latency 29.981 ms 2024-02-27T06:22:04.949 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:04 smithi131 ceph-mon[24834]: pgmap v257: 97 pgs: 97 active+clean; 202 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 805 op/s 2024-02-27T06:22:05.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:04 smithi150 ceph-mon[26806]: pgmap v257: 97 pgs: 97 active+clean; 202 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 805 op/s 2024-02-27T06:22:05.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 29003 20.25 MB/sec warmup 43 sec latency 21.489 ms 2024-02-27T06:22:06.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 29821 20.19 MB/sec warmup 44 sec latency 24.417 ms 2024-02-27T06:22:06.957 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:06 smithi131 ceph-mon[24834]: pgmap v258: 97 pgs: 97 active+clean; 207 MiB data, 817 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.0 MiB/s wr, 805 op/s 2024-02-27T06:22:07.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:06 smithi150 ceph-mon[26806]: pgmap v258: 97 pgs: 97 active+clean; 207 MiB data, 817 MiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.0 MiB/s wr, 805 op/s 2024-02-27T06:22:07.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 30560 20.01 MB/sec warmup 45 sec latency 20.956 ms 2024-02-27T06:22:08.493 INFO:teuthology.orchestra.run.smithi131.stdout: 5 31147 19.96 MB/sec warmup 46 sec latency 26.309 ms 2024-02-27T06:22:08.980 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:08 smithi131 ceph-mon[24834]: pgmap v259: 97 pgs: 97 active+clean; 209 MiB data, 865 MiB used, 714 GiB / 715 GiB avail; 9.9 KiB/s rd, 9.5 MiB/s wr, 879 op/s 2024-02-27T06:22:09.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:08 smithi150 ceph-mon[26806]: pgmap v259: 97 pgs: 97 active+clean; 209 MiB data, 865 MiB used, 714 GiB / 715 GiB avail; 9.9 KiB/s rd, 9.5 MiB/s wr, 879 op/s 2024-02-27T06:22:09.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 31849 20.19 MB/sec warmup 47 sec latency 21.173 ms 2024-02-27T06:22:10.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 32389 20.08 MB/sec warmup 48 sec latency 27.107 ms 2024-02-27T06:22:11.085 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:10 smithi131 ceph-mon[24834]: pgmap v260: 97 pgs: 97 active+clean; 215 MiB data, 891 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.1 MiB/s wr, 869 op/s 2024-02-27T06:22:11.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:10 smithi150 ceph-mon[26806]: pgmap v260: 97 pgs: 97 active+clean; 215 MiB data, 891 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.1 MiB/s wr, 869 op/s 2024-02-27T06:22:11.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 32916 19.81 MB/sec warmup 49 sec latency 25.064 ms 2024-02-27T06:22:12.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 33655 19.78 MB/sec warmup 50 sec latency 32.383 ms 2024-02-27T06:22:13.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:12 smithi131 ceph-mon[24834]: pgmap v261: 97 pgs: 97 active+clean; 226 MiB data, 927 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 883 op/s 2024-02-27T06:22:13.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:12 smithi150 ceph-mon[26806]: pgmap v261: 97 pgs: 97 active+clean; 226 MiB data, 927 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 883 op/s 2024-02-27T06:22:13.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 34576 19.83 MB/sec warmup 51 sec latency 13.177 ms 2024-02-27T06:22:14.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 35501 20.15 MB/sec warmup 52 sec latency 16.635 ms 2024-02-27T06:22:15.121 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:14 smithi131 ceph-mon[24834]: pgmap v262: 97 pgs: 97 active+clean; 222 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 914 op/s 2024-02-27T06:22:15.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:14 smithi150 ceph-mon[26806]: pgmap v262: 97 pgs: 97 active+clean; 222 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 914 op/s 2024-02-27T06:22:15.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 36130 20.04 MB/sec warmup 53 sec latency 25.225 ms 2024-02-27T06:22:16.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 36598 19.83 MB/sec warmup 54 sec latency 87.323 ms 2024-02-27T06:22:17.137 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:16 smithi131 ceph-mon[24834]: pgmap v263: 97 pgs: 97 active+clean; 228 MiB data, 981 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.4 MiB/s wr, 863 op/s 2024-02-27T06:22:17.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:16 smithi150 ceph-mon[26806]: pgmap v263: 97 pgs: 97 active+clean; 228 MiB data, 981 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.4 MiB/s wr, 863 op/s 2024-02-27T06:22:17.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 37063 19.66 MB/sec warmup 55 sec latency 115.033 ms 2024-02-27T06:22:18.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 37707 19.52 MB/sec warmup 56 sec latency 26.928 ms 2024-02-27T06:22:19.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:18 smithi131 ceph-mon[24834]: pgmap v264: 97 pgs: 97 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 922 op/s 2024-02-27T06:22:19.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:18 smithi150 ceph-mon[26806]: pgmap v264: 97 pgs: 97 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 922 op/s 2024-02-27T06:22:19.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 38417 19.58 MB/sec warmup 57 sec latency 25.018 ms 2024-02-27T06:22:20.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 39173 19.80 MB/sec warmup 58 sec latency 18.968 ms 2024-02-27T06:22:21.172 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:20 smithi131 ceph-mon[24834]: pgmap v265: 97 pgs: 97 active+clean; 231 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.5 MiB/s wr, 883 op/s 2024-02-27T06:22:21.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:20 smithi150 ceph-mon[26806]: pgmap v265: 97 pgs: 97 active+clean; 231 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.5 MiB/s wr, 883 op/s 2024-02-27T06:22:21.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 39751 19.65 MB/sec warmup 59 sec latency 22.504 ms 2024-02-27T06:22:22.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 40382 19.53 MB/sec warmup 60 sec latency 37.208 ms 2024-02-27T06:22:23.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:22 smithi150 ceph-mon[26806]: pgmap v266: 97 pgs: 97 active+clean; 237 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 878 op/s 2024-02-27T06:22:23.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:22:23.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:22:23.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:22 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:22:23.187 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:22 smithi131 ceph-mon[24834]: pgmap v266: 97 pgs: 97 active+clean; 237 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 878 op/s 2024-02-27T06:22:23.188 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:22:23.188 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:22:23.188 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:22 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:22:23.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 41066 19.45 MB/sec warmup 61 sec latency 23.825 ms 2024-02-27T06:22:24.192 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:23 smithi131 ceph-mon[24834]: Upgrade: Target is version 19.0.0-1736-g635ee685 (unknown) 2024-02-27T06:22:24.192 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:23 smithi131 ceph-mon[24834]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, digests ['quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e'] 2024-02-27T06:22:24.192 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:23 smithi131 ceph-mon[24834]: Upgrade: Need to upgrade myself (mgr.smithi131.frtgqo) 2024-02-27T06:22:24.192 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:23 smithi131 ceph-mon[24834]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e on smithi150 2024-02-27T06:22:24.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:23 smithi150 ceph-mon[26806]: Upgrade: Target is version 19.0.0-1736-g635ee685 (unknown) 2024-02-27T06:22:24.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:23 smithi150 ceph-mon[26806]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, digests ['quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e'] 2024-02-27T06:22:24.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:23 smithi150 ceph-mon[26806]: Upgrade: Need to upgrade myself (mgr.smithi131.frtgqo) 2024-02-27T06:22:24.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:23 smithi150 ceph-mon[26806]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e on smithi150 2024-02-27T06:22:24.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 41695 19.39 MB/sec warmup 62 sec latency 22.652 ms 2024-02-27T06:22:24.651 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:22:25.201 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:24 smithi131 ceph-mon[24834]: pgmap v267: 97 pgs: 97 active+clean; 242 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 770 op/s 2024-02-27T06:22:25.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:24 smithi150 ceph-mon[26806]: pgmap v267: 97 pgs: 97 active+clean; 242 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 770 op/s 2024-02-27T06:22:25.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 42389 19.56 MB/sec warmup 63 sec latency 22.787 ms 2024-02-27T06:22:25.580 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (6m) 65s ago 7m 22.8M - ba2b418f427c 9fbc60950b19 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (7m) 65s ago 7m 6405k - 17.2.0 e1d6a67b021e 5f66626dee40 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (6m) 67s ago 6m 6405k - 17.2.0 e1d6a67b021e e37d494b0a36 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (5m) 65s ago 7m 50.2M - 8.3.5 dad864ee21e9 ea727583c9f8 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (96s) 65s ago 95s 13.5M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (93s) 67s ago 93s 11.8M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:9283 running (8m) 65s ago 8m 462M - 17.2.0 e1d6a67b021e 8f815fe96db7 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283 running (6m) 67s ago 6m 412M - 17.2.0 e1d6a67b021e c959900ba7a2 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (8m) 65s ago 8m 46.4M 2048M 17.2.0 e1d6a67b021e 8e3c593dcb9f 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (6m) 67s ago 6m 37.5M 2048M 17.2.0 e1d6a67b021e 976f96599f98 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (75s) 65s ago 75s 51.3M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (70s) 67s ago 70s 48.8M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (7m) 65s ago 7m 17.2M - 1dbe0e931976 c20eeeeb0ea2 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (6m) 67s ago 6m 16.5M - 1dbe0e931976 26b7bdf144f9 2024-02-27T06:22:25.581 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (5m) 65s ago 5m 42.2M 3216M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:22:25.582 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (5m) 65s ago 5m 43.0M 3216M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:22:25.582 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (5m) 65s ago 5m 40.7M 3216M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:22:25.582 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (4m) 65s ago 4m 39.2M 3216M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:22:25.582 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (4m) 67s ago 4m 40.1M 3984M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:22:25.582 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (3m) 67s ago 3m 39.7M 3984M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:22:25.582 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (3m) 67s ago 3m 39.5M 3984M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:22:25.582 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (3m) 67s ago 3m 43.1M 3984M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:22:25.582 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (6m) 65s ago 6m 65.2M - 514e6a882f6e 3b3a90b3ef83 2024-02-27T06:22:26.183 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:22:26.183 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:22:26.183 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:22:26.184 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:22:26.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:26 smithi131 ceph-mon[24834]: from='client.24387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:22:26.211 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:22:26.211 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:26 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:22:26.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:26 smithi150 ceph-mon[26806]: from='client.24387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:22:26.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:22:26.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:26 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:22:26.494 INFO:teuthology.orchestra.run.smithi131.stdout: 5 42962 19.54 MB/sec warmup 64 sec latency 27.165 ms 2024-02-27T06:22:26.719 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:22:26.719 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:22:26.719 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:22:26.719 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [], 2024-02-27T06:22:26.719 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "0/23 daemons upgraded", 2024-02-27T06:22:26.719 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e image on host smithi150" 2024-02-27T06:22:26.719 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:22:27.215 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:27 smithi131 ceph-mon[24834]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:22:27.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:27 smithi131 ceph-mon[24834]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:22:27.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:27 smithi131 ceph-mon[24834]: pgmap v268: 97 pgs: 97 active+clean; 250 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.2 MiB/s wr, 906 op/s 2024-02-27T06:22:27.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:27 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2832359847' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:22:27.310 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:22:27.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:27 smithi150 ceph-mon[26806]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:22:27.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:27 smithi150 ceph-mon[26806]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:22:27.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:27 smithi150 ceph-mon[26806]: pgmap v268: 97 pgs: 97 active+clean; 250 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.2 MiB/s wr, 906 op/s 2024-02-27T06:22:27.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:27 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/2832359847' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:22:27.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 43465 19.34 MB/sec warmup 65 sec latency 26.276 ms 2024-02-27T06:22:28.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:28 smithi150 ceph-mon[26806]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:22:28.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:28 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1124901340' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:22:28.114 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:28 smithi131 ceph-mon[24834]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:22:28.114 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:28 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1124901340' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:22:28.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 44010 19.21 MB/sec warmup 66 sec latency 22.759 ms 2024-02-27T06:22:29.123 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:29 smithi131 ceph-mon[24834]: pgmap v269: 97 pgs: 97 active+clean; 252 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 881 op/s 2024-02-27T06:22:29.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:29 smithi150 ceph-mon[26806]: pgmap v269: 97 pgs: 97 active+clean; 252 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 881 op/s 2024-02-27T06:22:29.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 44480 19.08 MB/sec warmup 67 sec latency 40.952 ms 2024-02-27T06:22:30.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 44946 18.97 MB/sec warmup 68 sec latency 33.429 ms 2024-02-27T06:22:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:31 smithi131 ceph-mon[24834]: pgmap v270: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.1 MiB/s wr, 859 op/s 2024-02-27T06:22:31.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:31 smithi150 ceph-mon[26806]: pgmap v270: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.1 MiB/s wr, 859 op/s 2024-02-27T06:22:31.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 45392 18.91 MB/sec warmup 69 sec latency 29.908 ms 2024-02-27T06:22:32.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 45981 18.98 MB/sec warmup 70 sec latency 40.008 ms 2024-02-27T06:22:33.214 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:33 smithi131 ceph-mon[24834]: pgmap v271: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 9.0 MiB/s wr, 902 op/s 2024-02-27T06:22:33.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:33 smithi150 ceph-mon[26806]: pgmap v271: 97 pgs: 97 active+clean; 258 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 9.0 MiB/s wr, 902 op/s 2024-02-27T06:22:33.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 46396 18.95 MB/sec warmup 71 sec latency 34.895 ms 2024-02-27T06:22:34.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 46802 18.74 MB/sec warmup 72 sec latency 32.988 ms 2024-02-27T06:22:35.239 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:35 smithi131 ceph-mon[24834]: pgmap v272: 97 pgs: 97 active+clean; 260 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 9.1 MiB/s wr, 814 op/s 2024-02-27T06:22:35.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:35 smithi150 ceph-mon[26806]: pgmap v272: 97 pgs: 97 active+clean; 260 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.8 KiB/s rd, 9.1 MiB/s wr, 814 op/s 2024-02-27T06:22:35.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 47316 18.66 MB/sec warmup 73 sec latency 29.859 ms 2024-02-27T06:22:36.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 48153 18.62 MB/sec warmup 74 sec latency 20.443 ms 2024-02-27T06:22:37.257 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:37 smithi131 ceph-mon[24834]: pgmap v273: 97 pgs: 97 active+clean; 264 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.0 MiB/s wr, 777 op/s 2024-02-27T06:22:37.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:37 smithi150 ceph-mon[26806]: pgmap v273: 97 pgs: 97 active+clean; 264 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.0 MiB/s wr, 777 op/s 2024-02-27T06:22:37.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 48795 18.60 MB/sec warmup 75 sec latency 37.066 ms 2024-02-27T06:22:38.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 49370 18.65 MB/sec warmup 76 sec latency 66.902 ms 2024-02-27T06:22:39.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:39 smithi131 ceph-mon[24834]: pgmap v274: 97 pgs: 97 active+clean; 271 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 8.4 MiB/s wr, 742 op/s 2024-02-27T06:22:39.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:39 smithi150 ceph-mon[26806]: pgmap v274: 97 pgs: 97 active+clean; 271 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 8.4 MiB/s wr, 742 op/s 2024-02-27T06:22:39.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 49776 18.62 MB/sec warmup 77 sec latency 35.158 ms 2024-02-27T06:22:40.085 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:40 smithi131 ceph-mon[24834]: pgmap v275: 97 pgs: 97 active+clean; 275 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.0 MiB/s wr, 853 op/s 2024-02-27T06:22:40.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:40 smithi150 ceph-mon[26806]: pgmap v275: 97 pgs: 97 active+clean; 275 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.0 MiB/s wr, 853 op/s 2024-02-27T06:22:40.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 50167 18.50 MB/sec warmup 78 sec latency 39.435 ms 2024-02-27T06:22:41.496 INFO:teuthology.orchestra.run.smithi131.stdout: 5 50611 18.34 MB/sec warmup 79 sec latency 25.603 ms 2024-02-27T06:22:42.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 51336 18.33 MB/sec warmup 80 sec latency 20.757 ms 2024-02-27T06:22:43.107 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:42 smithi150 ceph-mon[26806]: pgmap v276: 97 pgs: 97 active+clean; 278 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.5 MiB/s wr, 772 op/s 2024-02-27T06:22:43.123 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:42 smithi131 ceph-mon[24834]: pgmap v276: 97 pgs: 97 active+clean; 278 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.5 MiB/s wr, 772 op/s 2024-02-27T06:22:43.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 51960 18.24 MB/sec warmup 81 sec latency 58.295 ms 2024-02-27T06:22:44.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 52721 18.35 MB/sec warmup 82 sec latency 32.295 ms 2024-02-27T06:22:45.141 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:44 smithi131 ceph-mon[24834]: pgmap v277: 97 pgs: 97 active+clean; 282 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.9 MiB/s wr, 701 op/s 2024-02-27T06:22:45.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:44 smithi150 ceph-mon[26806]: pgmap v277: 97 pgs: 97 active+clean; 282 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.9 MiB/s wr, 701 op/s 2024-02-27T06:22:45.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 53322 18.44 MB/sec warmup 83 sec latency 23.144 ms 2024-02-27T06:22:46.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 53846 18.33 MB/sec warmup 84 sec latency 23.862 ms 2024-02-27T06:22:47.105 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:46 smithi150 ceph-mon[26806]: pgmap v278: 97 pgs: 97 active+clean; 287 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 7.6 MiB/s wr, 773 op/s 2024-02-27T06:22:47.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:46 smithi131 ceph-mon[24834]: pgmap v278: 97 pgs: 97 active+clean; 287 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 7.6 MiB/s wr, 773 op/s 2024-02-27T06:22:47.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 54380 18.27 MB/sec warmup 85 sec latency 27.360 ms 2024-02-27T06:22:48.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 55057 18.18 MB/sec warmup 86 sec latency 31.498 ms 2024-02-27T06:22:49.036 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:48 smithi131 ceph-mon[24834]: pgmap v279: 97 pgs: 97 active+clean; 288 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 788 op/s 2024-02-27T06:22:49.042 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:48 smithi150 ceph-mon[26806]: pgmap v279: 97 pgs: 97 active+clean; 288 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 788 op/s 2024-02-27T06:22:49.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 55604 18.10 MB/sec warmup 87 sec latency 39.174 ms 2024-02-27T06:22:50.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 56327 18.20 MB/sec warmup 88 sec latency 40.446 ms 2024-02-27T06:22:51.057 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:50 smithi131 ceph-mon[24834]: pgmap v280: 97 pgs: 97 active+clean; 296 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.3 MiB/s wr, 803 op/s 2024-02-27T06:22:51.057 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:50 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:22:51.057 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:50 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vvgbyr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:22:51.057 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:50 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:22:51.057 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:50 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:22:51.107 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:50 smithi150 ceph-mon[26806]: pgmap v280: 97 pgs: 97 active+clean; 296 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 8.3 MiB/s wr, 803 op/s 2024-02-27T06:22:51.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:50 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:22:51.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:50 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vvgbyr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:22:51.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:50 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:22:51.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:50 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:22:51.495 INFO:teuthology.orchestra.run.smithi131.stdout: 5 56900 18.28 MB/sec warmup 89 sec latency 59.624 ms 2024-02-27T06:22:52.066 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:51 smithi131 ceph-mon[24834]: Upgrade: Updating mgr.smithi150.vvgbyr 2024-02-27T06:22:52.066 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:51 smithi131 ceph-mon[24834]: Deploying daemon mgr.smithi150.vvgbyr on smithi150 2024-02-27T06:22:52.069 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:51 smithi150 ceph-mon[26806]: Upgrade: Updating mgr.smithi150.vvgbyr 2024-02-27T06:22:52.069 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:51 smithi150 ceph-mon[26806]: Deploying daemon mgr.smithi150.vvgbyr on smithi150 2024-02-27T06:22:52.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 57376 18.16 MB/sec warmup 90 sec latency 46.412 ms 2024-02-27T06:22:53.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:52 smithi131 ceph-mon[24834]: pgmap v281: 97 pgs: 97 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 8.2 MiB/s wr, 772 op/s 2024-02-27T06:22:53.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:52 smithi150 ceph-mon[26806]: pgmap v281: 97 pgs: 97 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 8.2 MiB/s wr, 772 op/s 2024-02-27T06:22:53.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 57737 18.03 MB/sec warmup 91 sec latency 77.347 ms 2024-02-27T06:22:54.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 58231 18.00 MB/sec warmup 92 sec latency 112.673 ms 2024-02-27T06:22:55.087 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:54 smithi131 ceph-mon[24834]: pgmap v282: 97 pgs: 97 active+clean; 301 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 8.5 MiB/s wr, 735 op/s 2024-02-27T06:22:55.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:54 smithi150 ceph-mon[26806]: pgmap v282: 97 pgs: 97 active+clean; 301 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.5 KiB/s rd, 8.5 MiB/s wr, 735 op/s 2024-02-27T06:22:55.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 58905 17.89 MB/sec warmup 93 sec latency 112.356 ms 2024-02-27T06:22:56.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 59723 18.02 MB/sec warmup 94 sec latency 60.018 ms 2024-02-27T06:22:57.100 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:56 smithi131 ceph-mon[24834]: pgmap v283: 97 pgs: 97 active+clean; 305 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 811 op/s 2024-02-27T06:22:57.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:56 smithi150 ceph-mon[26806]: pgmap v283: 97 pgs: 97 active+clean; 305 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 811 op/s 2024-02-27T06:22:57.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 60449 18.14 MB/sec warmup 95 sec latency 29.871 ms 2024-02-27T06:22:57.728 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:22:58.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 60949 18.02 MB/sec warmup 96 sec latency 131.361 ms 2024-02-27T06:22:58.641 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:22:58.641 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (6m) 98s ago 7m 22.8M - ba2b418f427c 9fbc60950b19 2024-02-27T06:22:58.641 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (7m) 98s ago 7m 6405k - 17.2.0 e1d6a67b021e 5f66626dee40 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (7m) 100s ago 7m 6405k - 17.2.0 e1d6a67b021e e37d494b0a36 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (6m) 98s ago 7m 50.2M - 8.3.5 dad864ee21e9 ea727583c9f8 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (2m) 98s ago 2m 13.5M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (2m) 100s ago 2m 11.8M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:9283 running (9m) 98s ago 9m 462M - 17.2.0 e1d6a67b021e 8f815fe96db7 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283 running (7m) 100s ago 7m 412M - 17.2.0 e1d6a67b021e c959900ba7a2 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (9m) 98s ago 9m 46.4M 2048M 17.2.0 e1d6a67b021e 8e3c593dcb9f 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (7m) 100s ago 7m 37.5M 2048M 17.2.0 e1d6a67b021e 976f96599f98 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (109s) 98s ago 108s 51.3M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (103s) 100s ago 103s 48.8M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (7m) 98s ago 7m 17.2M - 1dbe0e931976 c20eeeeb0ea2 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (7m) 100s ago 7m 16.5M - 1dbe0e931976 26b7bdf144f9 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (6m) 98s ago 6m 42.2M 3216M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:22:58.642 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (6m) 98s ago 6m 43.0M 3216M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:22:58.643 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (5m) 98s ago 5m 40.7M 3216M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:22:58.643 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (5m) 98s ago 5m 39.2M 3216M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:22:58.643 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (4m) 100s ago 4m 40.1M 3984M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:22:58.643 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (4m) 100s ago 4m 39.7M 3984M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:22:58.643 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (4m) 100s ago 4m 39.5M 3984M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:22:58.643 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (3m) 100s ago 3m 43.1M 3984M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:22:58.643 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (6m) 98s ago 6m 65.2M - 514e6a882f6e 3b3a90b3ef83 2024-02-27T06:22:59.003 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:58 smithi131 ceph-mon[24834]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:22:59.003 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:58 smithi131 ceph-mon[24834]: pgmap v284: 97 pgs: 97 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 8.8 MiB/s wr, 783 op/s 2024-02-27T06:22:59.162 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:22:59.163 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:22:59.163 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:22:59.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:22:59.163 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:22:59.163 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:22:59.163 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:22:59.164 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:22:59.164 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T06:22:59.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:22:59.164 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:22:59.164 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:22:59.164 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:22:59.165 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:22:59.165 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-02-27T06:22:59.165 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:22:59.165 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:22:59.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:58 smithi150 ceph-mon[26806]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:22:59.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:58 smithi150 ceph-mon[26806]: pgmap v284: 97 pgs: 97 active+clean; 304 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.0 KiB/s rd, 8.8 MiB/s wr, 783 op/s 2024-02-27T06:22:59.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 61445 17.98 MB/sec warmup 97 sec latency 27.067 ms 2024-02-27T06:22:59.604 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:22:59.604 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:22:59.604 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:22:59.604 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [], 2024-02-27T06:22:59.604 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "0/23 daemons upgraded", 2024-02-27T06:22:59.604 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading mgr daemons" 2024-02-27T06:22:59.604 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:23:00.015 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:59 smithi131 ceph-mon[24834]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:00.015 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:59 smithi131 ceph-mon[24834]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:00.015 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:22:59 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1937629938' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:23:00.136 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:23:00.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:59 smithi150 ceph-mon[26806]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:00.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:59 smithi150 ceph-mon[26806]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:00.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:22:59 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1937629938' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:23:00.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 62013 17.88 MB/sec warmup 98 sec latency 40.800 ms 2024-02-27T06:23:01.021 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:00 smithi131 ceph-mon[24834]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:01.021 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:00 smithi131 ceph-mon[24834]: pgmap v285: 97 pgs: 97 active+clean; 313 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.7 MiB/s wr, 879 op/s 2024-02-27T06:23:01.021 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:00 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2426425518' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:23:01.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:00 smithi150 ceph-mon[26806]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:01.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:00 smithi150 ceph-mon[26806]: pgmap v285: 97 pgs: 97 active+clean; 313 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.7 MiB/s wr, 879 op/s 2024-02-27T06:23:01.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:00 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/2426425518' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:23:01.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 62745 17.90 MB/sec warmup 99 sec latency 31.571 ms 2024-02-27T06:23:02.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 63539 18.01 MB/sec warmup 100 sec latency 22.036 ms 2024-02-27T06:23:03.031 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:02 smithi131 ceph-mon[24834]: pgmap v286: 97 pgs: 97 active+clean; 323 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.6 MiB/s wr, 893 op/s 2024-02-27T06:23:03.355 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:02 smithi150 ceph-mon[26806]: pgmap v286: 97 pgs: 97 active+clean; 323 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.6 MiB/s wr, 893 op/s 2024-02-27T06:23:03.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 64025 18.00 MB/sec warmup 101 sec latency 51.158 ms 2024-02-27T06:23:04.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 64534 17.90 MB/sec warmup 102 sec latency 79.848 ms 2024-02-27T06:23:05.061 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:04 smithi131 ceph-mon[24834]: pgmap v287: 97 pgs: 97 active+clean; 324 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.7 MiB/s wr, 783 op/s 2024-02-27T06:23:05.061 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:04 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:23:05.061 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:04 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:23:05.061 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:04 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:23:05.061 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:04 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:23:05.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:04 smithi150 ceph-mon[26806]: pgmap v287: 97 pgs: 97 active+clean; 324 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 8.7 MiB/s wr, 783 op/s 2024-02-27T06:23:05.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:04 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:23:05.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:04 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:23:05.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:04 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:23:05.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:04 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:23:05.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 64920 17.83 MB/sec warmup 103 sec latency 57.277 ms 2024-02-27T06:23:06.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 65682 17.78 MB/sec warmup 104 sec latency 29.603 ms 2024-02-27T06:23:07.084 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:06 smithi131 ceph-mon[24834]: pgmap v288: 97 pgs: 97 active+clean; 323 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.0 MiB/s wr, 767 op/s 2024-02-27T06:23:07.271 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:06 smithi150 ceph-mon[26806]: pgmap v288: 97 pgs: 97 active+clean; 323 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.0 MiB/s wr, 767 op/s 2024-02-27T06:23:07.498 INFO:teuthology.orchestra.run.smithi131.stdout: 5 66317 17.80 MB/sec warmup 105 sec latency 24.473 ms 2024-02-27T06:23:08.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 67132 17.90 MB/sec warmup 106 sec latency 23.581 ms 2024-02-27T06:23:09.059 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:08 smithi131 ceph-mon[24834]: pgmap v289: 97 pgs: 97 active+clean; 333 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 9.6 MiB/s wr, 902 op/s 2024-02-27T06:23:09.059 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:08 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:23:09.229 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:08 smithi150 ceph-mon[26806]: pgmap v289: 97 pgs: 97 active+clean; 333 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 9.6 MiB/s wr, 902 op/s 2024-02-27T06:23:09.229 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:08 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:23:09.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 67573 17.89 MB/sec warmup 107 sec latency 28.620 ms 2024-02-27T06:23:10.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 67935 17.76 MB/sec warmup 108 sec latency 188.713 ms 2024-02-27T06:23:10.606 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:10 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:23:10.606 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:10 smithi150 ceph-mon[26806]: pgmap v290: 97 pgs: 97 active+clean; 341 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 926 op/s 2024-02-27T06:23:10.645 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:10 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:23:10.645 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:10 smithi131 ceph-mon[24834]: pgmap v290: 97 pgs: 97 active+clean; 341 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 926 op/s 2024-02-27T06:23:11.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 68324 17.68 MB/sec warmup 109 sec latency 29.785 ms 2024-02-27T06:23:12.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 68886 17.66 MB/sec warmup 110 sec latency 26.838 ms 2024-02-27T06:23:13.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:12 smithi150 ceph-mon[26806]: pgmap v291: 97 pgs: 97 active+clean; 340 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 812 op/s 2024-02-27T06:23:13.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:12 smithi150 ceph-mon[26806]: Standby manager daemon smithi150.vvgbyr restarted 2024-02-27T06:23:13.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:12 smithi150 ceph-mon[26806]: Standby manager daemon smithi150.vvgbyr started 2024-02-27T06:23:13.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:12 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/crt"}]: dispatch 2024-02-27T06:23:13.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:12 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:23:13.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:12 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/key"}]: dispatch 2024-02-27T06:23:13.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:12 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:23:13.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:12 smithi131 ceph-mon[24834]: pgmap v291: 97 pgs: 97 active+clean; 340 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 812 op/s 2024-02-27T06:23:13.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:12 smithi131 ceph-mon[24834]: Standby manager daemon smithi150.vvgbyr restarted 2024-02-27T06:23:13.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:12 smithi131 ceph-mon[24834]: Standby manager daemon smithi150.vvgbyr started 2024-02-27T06:23:13.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:12 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/crt"}]: dispatch 2024-02-27T06:23:13.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:12 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:23:13.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:12 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/key"}]: dispatch 2024-02-27T06:23:13.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:12 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:23:13.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 69509 17.57 MB/sec warmup 111 sec latency 34.270 ms 2024-02-27T06:23:13.934 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:13 smithi131 ceph-mon[24834]: mgrmap e22: smithi131.frtgqo(active, since 7m), standbys: smithi150.vvgbyr 2024-02-27T06:23:13.934 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:13 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:23:13.934 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:13 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:23:13.934 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:13 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:23:13.934 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:13 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:23:13.934 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:13 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr fail", "who": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:23:13.935 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:13 smithi131 ceph-mon[24834]: osdmap e71: 8 total, 8 up, 8 in 2024-02-27T06:23:14.350 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:13 smithi150 ceph-mon[26806]: mgrmap e22: smithi131.frtgqo(active, since 7m), standbys: smithi150.vvgbyr 2024-02-27T06:23:14.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:13 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:23:14.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:13 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:23:14.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:13 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' 2024-02-27T06:23:14.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:13 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:23:14.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:13 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr fail", "who": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:23:14.351 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:13 smithi150 ceph-mon[26806]: osdmap e71: 8 total, 8 up, 8 in 2024-02-27T06:23:14.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 70298 17.68 MB/sec warmup 112 sec latency 27.747 ms 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: Upgrade: Need to upgrade myself (mgr.smithi131.frtgqo) 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: Failing over to other MGR 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: pgmap v293: 97 pgs: 97 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.2 MiB/s wr, 821 op/s 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mgr metadata", "who": "smithi150.vvgbyr", "id": "smithi150.vvgbyr"}]: dispatch 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "mgr fail", "who": "smithi131.frtgqo"}]': finished 2024-02-27T06:23:14.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: mgrmap e23: smithi150.vvgbyr(active, starting, since 0.996264s) 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: Manager daemon smithi150.vvgbyr is now available 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:23:14.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi150.vvgbyr/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:23:14.941 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi150.vvgbyr/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:23:14.941 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi150.vvgbyr/trash_purge_schedule"}]: dispatch 2024-02-27T06:23:14.941 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi150.vvgbyr/trash_purge_schedule"}]: dispatch 2024-02-27T06:23:14.941 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:14.941 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:14 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:15.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: Upgrade: Need to upgrade myself (mgr.smithi131.frtgqo) 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: Failing over to other MGR 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: pgmap v293: 97 pgs: 97 active+clean; 340 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 8.2 MiB/s wr, 821 op/s 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mgr metadata", "who": "smithi150.vvgbyr", "id": "smithi150.vvgbyr"}]: dispatch 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.14202 172.21.15.131:0/2657032103' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "mgr fail", "who": "smithi131.frtgqo"}]': finished 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: mgrmap e23: smithi150.vvgbyr(active, starting, since 0.996264s) 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:23:15.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: Manager daemon smithi150.vvgbyr is now available 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi150.vvgbyr/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi150.vvgbyr/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi150.vvgbyr/trash_purge_schedule"}]: dispatch 2024-02-27T06:23:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi150.vvgbyr/trash_purge_schedule"}]: dispatch 2024-02-27T06:23:15.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:15.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:14 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:15.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 70912 17.71 MB/sec warmup 113 sec latency 20.160 ms 2024-02-27T06:23:16.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:16 smithi131 ceph-mon[24834]: Deploying cephadm binary to smithi150 2024-02-27T06:23:16.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:16 smithi131 ceph-mon[24834]: Deploying cephadm binary to smithi131 2024-02-27T06:23:16.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:16 smithi131 ceph-mon[24834]: mgrmap e24: smithi150.vvgbyr(active, since 2s) 2024-02-27T06:23:16.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:16 smithi131 ceph-mon[24834]: pgmap v3: 97 pgs: 97 active+clean; 341 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-02-27T06:23:16.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 71446 17.66 MB/sec warmup 114 sec latency 30.063 ms 2024-02-27T06:23:16.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:16 smithi150 ceph-mon[26806]: Deploying cephadm binary to smithi150 2024-02-27T06:23:16.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:16 smithi150 ceph-mon[26806]: Deploying cephadm binary to smithi131 2024-02-27T06:23:16.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:16 smithi150 ceph-mon[26806]: mgrmap e24: smithi150.vvgbyr(active, since 2s) 2024-02-27T06:23:16.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:16 smithi150 ceph-mon[26806]: pgmap v3: 97 pgs: 97 active+clean; 341 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-02-27T06:23:17.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:17 smithi131 ceph-mon[24834]: [27/Feb/2024:06:23:15] ENGINE Bus STARTING 2024-02-27T06:23:17.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:17 smithi131 ceph-mon[24834]: [27/Feb/2024:06:23:15] ENGINE Serving on http://172.21.15.150:8765 2024-02-27T06:23:17.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:17 smithi131 ceph-mon[24834]: [27/Feb/2024:06:23:16] ENGINE Serving on https://172.21.15.150:7150 2024-02-27T06:23:17.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:17 smithi131 ceph-mon[24834]: [27/Feb/2024:06:23:16] ENGINE Bus STARTED 2024-02-27T06:23:17.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:17 smithi131 ceph-mon[24834]: pgmap v4: 97 pgs: 97 active+clean; 341 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-02-27T06:23:17.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 71976 17.63 MB/sec warmup 115 sec latency 27.946 ms 2024-02-27T06:23:17.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:17 smithi150 ceph-mon[26806]: [27/Feb/2024:06:23:15] ENGINE Bus STARTING 2024-02-27T06:23:17.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:17 smithi150 ceph-mon[26806]: [27/Feb/2024:06:23:15] ENGINE Serving on http://172.21.15.150:8765 2024-02-27T06:23:17.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:17 smithi150 ceph-mon[26806]: [27/Feb/2024:06:23:16] ENGINE Serving on https://172.21.15.150:7150 2024-02-27T06:23:17.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:17 smithi150 ceph-mon[26806]: [27/Feb/2024:06:23:16] ENGINE Bus STARTED 2024-02-27T06:23:17.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:17 smithi150 ceph-mon[26806]: pgmap v4: 97 pgs: 97 active+clean; 341 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-02-27T06:23:18.470 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:18 smithi131 ceph-mon[24834]: mgrmap e25: smithi150.vvgbyr(active, since 4s) 2024-02-27T06:23:18.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 72663 17.58 MB/sec warmup 116 sec latency 33.523 ms 2024-02-27T06:23:18.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:18 smithi150 ceph-mon[26806]: mgrmap e25: smithi150.vvgbyr(active, since 4s) 2024-02-27T06:23:19.489 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:19 smithi131 ceph-mon[24834]: pgmap v5: 97 pgs: 97 active+clean; 341 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-02-27T06:23:19.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 73223 17.54 MB/sec warmup 117 sec latency 30.769 ms 2024-02-27T06:23:19.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:19 smithi150 ceph-mon[26806]: pgmap v5: 97 pgs: 97 active+clean; 341 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-02-27T06:23:20.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 73836 17.58 MB/sec warmup 118 sec latency 31.600 ms 2024-02-27T06:23:21.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 74411 17.62 MB/sec warmup 119 sec latency 23.832 ms 2024-02-27T06:23:21.524 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:21 smithi131 ceph-mon[24834]: pgmap v6: 97 pgs: 97 active+clean; 341 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-02-27T06:23:21.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:21 smithi150 ceph-mon[26806]: pgmap v6: 97 pgs: 97 active+clean; 341 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-02-27T06:23:23.032 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:22 smithi131 ceph-mon[24834]: pgmap v7: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 5.8 MiB/s wr, 601 op/s 2024-02-27T06:23:23.098 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:22 smithi150 ceph-mon[26806]: pgmap v7: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 5.8 MiB/s wr, 601 op/s 2024-02-27T06:23:23.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 75346 9.46 MB/sec execute 1 sec latency 30.031 ms 2024-02-27T06:23:24.040 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:23 smithi131 ceph-mon[24834]: Standby manager daemon smithi131.frtgqo started 2024-02-27T06:23:24.040 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:23 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.131:0/946504245' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi131.frtgqo/crt"}]: dispatch 2024-02-27T06:23:24.041 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:23 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.131:0/946504245' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:23:24.041 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:23 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.131:0/946504245' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi131.frtgqo/key"}]: dispatch 2024-02-27T06:23:24.041 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:23 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.131:0/946504245' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:23:24.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:23 smithi150 ceph-mon[26806]: Standby manager daemon smithi131.frtgqo started 2024-02-27T06:23:24.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:23 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.131:0/946504245' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi131.frtgqo/crt"}]: dispatch 2024-02-27T06:23:24.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:23 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.131:0/946504245' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:23:24.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:23 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.131:0/946504245' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi131.frtgqo/key"}]: dispatch 2024-02-27T06:23:24.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:23 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.131:0/946504245' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:23:24.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 75855 12.13 MB/sec execute 2 sec latency 29.392 ms 2024-02-27T06:23:25.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:24 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mgr metadata", "who": "smithi131.frtgqo", "id": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:23:25.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:24 smithi131 ceph-mon[24834]: mgrmap e26: smithi150.vvgbyr(active, since 10s), standbys: smithi131.frtgqo 2024-02-27T06:23:25.048 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:24 smithi131 ceph-mon[24834]: pgmap v8: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 4.5 MiB/s wr, 467 op/s 2024-02-27T06:23:25.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:24 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mgr metadata", "who": "smithi131.frtgqo", "id": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:23:25.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:24 smithi150 ceph-mon[26806]: mgrmap e26: smithi150.vvgbyr(active, since 10s), standbys: smithi131.frtgqo 2024-02-27T06:23:25.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:24 smithi150 ceph-mon[26806]: pgmap v8: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 4.5 MiB/s wr, 467 op/s 2024-02-27T06:23:25.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 76420 10.29 MB/sec execute 3 sec latency 37.029 ms 2024-02-27T06:23:26.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:26 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:26.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:26 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:26.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 76873 10.79 MB/sec execute 4 sec latency 29.477 ms 2024-02-27T06:23:26.606 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:26 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:26.606 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:26 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:27.499 INFO:teuthology.orchestra.run.smithi131.stdout: 5 77505 13.78 MB/sec execute 5 sec latency 47.939 ms 2024-02-27T06:23:27.586 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:27 smithi131 ceph-mon[24834]: pgmap v9: 97 pgs: 97 active+clean; 322 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 7.0 MiB/s wr, 717 op/s 2024-02-27T06:23:27.586 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:27 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:27.586 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:27 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:27.787 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:27 smithi150 ceph-mon[26806]: pgmap v9: 97 pgs: 97 active+clean; 322 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 7.0 MiB/s wr, 717 op/s 2024-02-27T06:23:27.788 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:27 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:27.788 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:27 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:28.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 78037 14.48 MB/sec execute 6 sec latency 29.029 ms 2024-02-27T06:23:29.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 78538 14.03 MB/sec execute 7 sec latency 35.409 ms 2024-02-27T06:23:29.520 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:29 smithi131 ceph-mon[24834]: pgmap v10: 97 pgs: 97 active+clean; 322 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 656 op/s 2024-02-27T06:23:29.520 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:29 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:23:29.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:29 smithi150 ceph-mon[26806]: pgmap v10: 97 pgs: 97 active+clean; 322 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 656 op/s 2024-02-27T06:23:29.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:29 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:23:30.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 78976 13.43 MB/sec execute 8 sec latency 32.149 ms 2024-02-27T06:23:30.807 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:23:31.032 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:31.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:31.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: pgmap v11: 97 pgs: 97 active+clean; 322 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 656 op/s 2024-02-27T06:23:31.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:31.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:31.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.034 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.034 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:30 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:31.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:31.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:31.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: pgmap v11: 97 pgs: 97 active+clean; 322 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 656 op/s 2024-02-27T06:23:31.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:31.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:31.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:31.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:30 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:31.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 79392 13.31 MB/sec execute 9 sec latency 53.660 ms 2024-02-27T06:23:31.948 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:23:31.948 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (7m) 1s ago 8m 25.3M - 0.23.0 ba2b418f427c 9fbc60950b19 2024-02-27T06:23:31.948 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (8m) 1s ago 8m 6405k - 17.2.0 e1d6a67b021e 5f66626dee40 2024-02-27T06:23:31.948 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (7m) 6s ago 7m 6405k - 17.2.0 e1d6a67b021e e37d494b0a36 2024-02-27T06:23:31.948 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (7m) 1s ago 8m 52.6M - 8.3.5 dad864ee21e9 ea727583c9f8 2024-02-27T06:23:31.948 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (2m) 1s ago 2m 37.5M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (2m) 6s ago 2m 12.6M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:9283 running (9m) 1s ago 9m 347M - 17.2.0 e1d6a67b021e 8f815fe96db7 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283 running (31s) 6s ago 7m 489M - 19.0.0-1736-g635ee685 c4eb0759a73a 4e3883537bf3 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (9m) 1s ago 9m 50.7M 2048M 17.2.0 e1d6a67b021e 8e3c593dcb9f 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (7m) 6s ago 7m 39.6M 2048M 17.2.0 e1d6a67b021e 976f96599f98 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (2m) 1s ago 2m 252M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (2m) 6s ago 2m 73.5M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (8m) 1s ago 8m 20.3M - 1.3.1 1dbe0e931976 c20eeeeb0ea2 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (7m) 6s ago 7m 19.7M - 1.3.1 1dbe0e931976 26b7bdf144f9 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (7m) 1s ago 7m 296M 1936M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (6m) 1s ago 6m 237M 1936M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (6m) 1s ago 6m 187M 1936M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:23:31.949 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (5m) 1s ago 5m 257M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:23:31.950 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (5m) 6s ago 5m 275M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:23:31.950 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (4m) 6s ago 4m 313M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:23:31.950 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (4m) 6s ago 4m 265M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:23:31.950 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (4m) 6s ago 4m 386M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:23:31.950 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (7m) 1s ago 7m 65.3M - 2.33.4 514e6a882f6e 3b3a90b3ef83 2024-02-27T06:23:32.038 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: Adjusting osd_memory_target on smithi150 to 2704M 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:32.040 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:23:32.040 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:32 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:23:32.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='client.14698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:32.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: Adjusting osd_memory_target on smithi150 to 2704M 2024-02-27T06:23:32.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:32.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:32.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:32.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-27T06:23:32.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:32.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:23:32.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:32 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:23:32.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 79948 12.65 MB/sec execute 10 sec latency 36.923 ms 2024-02-27T06:23:32.613 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:23:32.613 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:23:32.613 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:23:32.613 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:23:32.613 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:23:32.613 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-27T06:23:32.613 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T06:23:32.613 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:23:32.613 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:23:32.613 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T06:23:32.613 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:23:32.614 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:23:32.614 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:23:32.614 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:23:32.614 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:23:32.614 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-27T06:23:32.614 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T06:23:32.614 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:23:32.614 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:23:33.049 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: Adjusting osd_memory_target on smithi131 to 1936M 2024-02-27T06:23:33.049 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: Updating smithi131:/etc/ceph/ceph.conf 2024-02-27T06:23:33.050 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-27T06:23:33.050 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:33.050 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: Updating smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T06:23:33.050 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: Updating smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T06:23:33.050 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:23:33.050 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: pgmap v12: 97 pgs: 97 active+clean; 334 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 10 MiB/s wr, 968 op/s 2024-02-27T06:23:33.050 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:23:33.050 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: Updating smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T06:23:33.050 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/3314483057' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:23:33.050 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:33.050 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:33 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:23:33.072 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [], 2024-02-27T06:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "1/23 daemons upgraded", 2024-02-27T06:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "", 2024-02-27T06:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:23:33.073 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:23:33.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: Adjusting osd_memory_target on smithi131 to 1936M 2024-02-27T06:23:33.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: Updating smithi131:/etc/ceph/ceph.conf 2024-02-27T06:23:33.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-27T06:23:33.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:33.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: Updating smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T06:23:33.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: Updating smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T06:23:33.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:23:33.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: pgmap v12: 97 pgs: 97 active+clean; 334 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 10 MiB/s wr, 968 op/s 2024-02-27T06:23:33.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:23:33.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: Updating smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T06:23:33.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/3314483057' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:23:33.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:33.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:33 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:33.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 80338 12.22 MB/sec execute 11 sec latency 29.852 ms 2024-02-27T06:23:33.647 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:23:34.055 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: Updating smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T06:23:34.055 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.055 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:34.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: Reconfiguring alertmanager.smithi131 (dependencies changed)... 2024-02-27T06:23:34.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: Reconfiguring daemon alertmanager.smithi131 on smithi131 2024-02-27T06:23:34.056 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:34 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1669979072' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:23:34.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: Updating smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T06:23:34.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: from='client.14714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:23:34.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:34.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: Reconfiguring alertmanager.smithi131 (dependencies changed)... 2024-02-27T06:23:34.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: Reconfiguring daemon alertmanager.smithi131 on smithi131 2024-02-27T06:23:34.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:34 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1669979072' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:23:34.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 80932 13.35 MB/sec execute 12 sec latency 36.225 ms 2024-02-27T06:23:35.070 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:35 smithi131 ceph-mon[24834]: pgmap v13: 97 pgs: 97 active+clean; 334 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.6 MiB/s wr, 618 op/s 2024-02-27T06:23:35.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:35 smithi150 ceph-mon[26806]: pgmap v13: 97 pgs: 97 active+clean; 334 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.6 MiB/s wr, 618 op/s 2024-02-27T06:23:35.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 81487 13.98 MB/sec execute 13 sec latency 27.965 ms 2024-02-27T06:23:36.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 81834 13.62 MB/sec execute 14 sec latency 32.276 ms 2024-02-27T06:23:37.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 82347 13.25 MB/sec execute 15 sec latency 34.776 ms 2024-02-27T06:23:37.599 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:37 smithi131 ceph-mon[24834]: pgmap v14: 97 pgs: 97 active+clean; 325 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 903 op/s 2024-02-27T06:23:37.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:37 smithi150 ceph-mon[26806]: pgmap v14: 97 pgs: 97 active+clean; 325 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 903 op/s 2024-02-27T06:23:38.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 82739 13.23 MB/sec execute 16 sec latency 27.357 ms 2024-02-27T06:23:39.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 83259 12.92 MB/sec execute 17 sec latency 45.472 ms 2024-02-27T06:23:39.627 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:39 smithi131 ceph-mon[24834]: pgmap v15: 97 pgs: 97 active+clean; 325 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.5 MiB/s wr, 596 op/s 2024-02-27T06:23:39.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:39 smithi150 ceph-mon[26806]: pgmap v15: 97 pgs: 97 active+clean; 325 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.5 MiB/s wr, 596 op/s 2024-02-27T06:23:40.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 83923 12.83 MB/sec execute 18 sec latency 23.138 ms 2024-02-27T06:23:41.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 84747 13.96 MB/sec execute 19 sec latency 34.283 ms 2024-02-27T06:23:41.645 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:41 smithi131 ceph-mon[24834]: pgmap v16: 97 pgs: 97 active+clean; 325 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.5 MiB/s wr, 596 op/s 2024-02-27T06:23:41.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:41 smithi150 ceph-mon[26806]: pgmap v16: 97 pgs: 97 active+clean; 325 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 6.5 MiB/s wr, 596 op/s 2024-02-27T06:23:42.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 85266 14.10 MB/sec execute 20 sec latency 44.076 ms 2024-02-27T06:23:43.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:42 smithi150 ceph-mon[26806]: pgmap v17: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 907 op/s 2024-02-27T06:23:43.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:42 smithi131 ceph-mon[24834]: pgmap v17: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 907 op/s 2024-02-27T06:23:43.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 85790 13.84 MB/sec execute 21 sec latency 27.386 ms 2024-02-27T06:23:44.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 86278 13.97 MB/sec execute 22 sec latency 27.619 ms 2024-02-27T06:23:45.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:45 smithi131 ceph-mon[24834]: pgmap v18: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.8 MiB/s wr, 595 op/s 2024-02-27T06:23:45.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:45 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:23:45.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 86875 13.72 MB/sec execute 23 sec latency 78.485 ms 2024-02-27T06:23:45.808 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:45 smithi150 ceph-mon[26806]: pgmap v18: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.8 MiB/s wr, 595 op/s 2024-02-27T06:23:45.808 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:45 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:23:46.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 87349 13.53 MB/sec execute 24 sec latency 49.553 ms 2024-02-27T06:23:47.481 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:47 smithi131 ceph-mon[24834]: pgmap v19: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 931 op/s 2024-02-27T06:23:47.500 INFO:teuthology.orchestra.run.smithi131.stdout: 5 87848 13.79 MB/sec execute 25 sec latency 46.687 ms 2024-02-27T06:23:47.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:47 smithi150 ceph-mon[26806]: pgmap v19: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 931 op/s 2024-02-27T06:23:48.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 88569 14.21 MB/sec execute 26 sec latency 28.590 ms 2024-02-27T06:23:49.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 88959 14.15 MB/sec execute 27 sec latency 40.812 ms 2024-02-27T06:23:49.528 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:49 smithi131 ceph-mon[24834]: pgmap v20: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 6.4 MiB/s wr, 647 op/s 2024-02-27T06:23:49.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:49 smithi150 ceph-mon[26806]: pgmap v20: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 6.4 MiB/s wr, 647 op/s 2024-02-27T06:23:50.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 89399 13.91 MB/sec execute 28 sec latency 46.848 ms 2024-02-27T06:23:51.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 89704 13.65 MB/sec execute 29 sec latency 36.525 ms 2024-02-27T06:23:51.610 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:51 smithi131 ceph-mon[24834]: pgmap v21: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 6.4 MiB/s wr, 647 op/s 2024-02-27T06:23:51.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:51 smithi150 ceph-mon[26806]: pgmap v21: 97 pgs: 97 active+clean; 326 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 6.4 MiB/s wr, 647 op/s 2024-02-27T06:23:52.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 90133 13.66 MB/sec execute 30 sec latency 44.246 ms 2024-02-27T06:23:52.621 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:52 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:52.621 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:52 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:52.622 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:52 smithi131 ceph-mon[24834]: Reconfiguring grafana.smithi131 (dependencies changed)... 2024-02-27T06:23:52.622 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:52 smithi131 ceph-mon[24834]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-27T06:23:52.622 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:52 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:52.622 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:52 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:52.622 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:52 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T06:23:52.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:52 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:52.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:52 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:52.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:52 smithi150 ceph-mon[26806]: Reconfiguring grafana.smithi131 (dependencies changed)... 2024-02-27T06:23:52.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:52 smithi150 ceph-mon[26806]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-27T06:23:52.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:52 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:52.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:52 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:23:52.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:52 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T06:23:53.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 90781 13.43 MB/sec execute 31 sec latency 44.292 ms 2024-02-27T06:23:53.628 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:53 smithi131 ceph-mon[24834]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T06:23:53.628 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:53 smithi131 ceph-mon[24834]: Reconfiguring daemon grafana.smithi131 on smithi131 2024-02-27T06:23:53.628 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:53 smithi131 ceph-mon[24834]: pgmap v22: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 952 op/s 2024-02-27T06:23:53.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:53 smithi150 ceph-mon[26806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-27T06:23:53.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:53 smithi150 ceph-mon[26806]: Reconfiguring daemon grafana.smithi131 on smithi131 2024-02-27T06:23:53.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:53 smithi150 ceph-mon[26806]: pgmap v22: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 952 op/s 2024-02-27T06:23:54.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 91248 13.66 MB/sec execute 32 sec latency 35.312 ms 2024-02-27T06:23:55.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 91901 13.88 MB/sec execute 33 sec latency 44.740 ms 2024-02-27T06:23:55.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:55 smithi131 ceph-mon[24834]: pgmap v23: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.9 MiB/s wr, 641 op/s 2024-02-27T06:23:55.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:55 smithi150 ceph-mon[26806]: pgmap v23: 97 pgs: 97 active+clean; 330 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.9 MiB/s wr, 641 op/s 2024-02-27T06:23:56.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 92314 13.90 MB/sec execute 34 sec latency 35.588 ms 2024-02-27T06:23:57.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 92910 13.85 MB/sec execute 35 sec latency 31.490 ms 2024-02-27T06:23:57.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:57 smithi131 ceph-mon[24834]: pgmap v24: 97 pgs: 97 active+clean; 326 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 935 op/s 2024-02-27T06:23:57.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:57 smithi150 ceph-mon[26806]: pgmap v24: 97 pgs: 97 active+clean; 326 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.8 MiB/s wr, 935 op/s 2024-02-27T06:23:58.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 93458 13.94 MB/sec execute 36 sec latency 38.229 ms 2024-02-27T06:23:59.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 94145 13.76 MB/sec execute 37 sec latency 16.869 ms 2024-02-27T06:23:59.686 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:59 smithi131 ceph-mon[24834]: pgmap v25: 97 pgs: 97 active+clean; 326 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 599 op/s 2024-02-27T06:23:59.687 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:23:59 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:23:59.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:59 smithi150 ceph-mon[26806]: pgmap v25: 97 pgs: 97 active+clean; 326 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 599 op/s 2024-02-27T06:23:59.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:23:59 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:24:00.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 94531 13.61 MB/sec execute 38 sec latency 36.640 ms 2024-02-27T06:24:01.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 95156 13.93 MB/sec execute 39 sec latency 40.073 ms 2024-02-27T06:24:01.728 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:01 smithi131 ceph-mon[24834]: pgmap v26: 97 pgs: 97 active+clean; 326 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 599 op/s 2024-02-27T06:24:01.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:01 smithi150 ceph-mon[26806]: pgmap v26: 97 pgs: 97 active+clean; 326 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 599 op/s 2024-02-27T06:24:02.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 95660 14.02 MB/sec execute 40 sec latency 38.108 ms 2024-02-27T06:24:02.994 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:02 smithi131 ceph-mon[24834]: pgmap v27: 97 pgs: 97 active+clean; 323 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 930 op/s 2024-02-27T06:24:03.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:02 smithi150 ceph-mon[26806]: pgmap v27: 97 pgs: 97 active+clean; 323 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.5 MiB/s wr, 930 op/s 2024-02-27T06:24:03.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 95970 13.96 MB/sec execute 41 sec latency 31.327 ms 2024-02-27T06:24:04.266 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:24:04.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 96374 13.74 MB/sec execute 42 sec latency 46.905 ms 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 starting - - - - 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (9m) 35s ago 9m 6405k - 17.2.0 e1d6a67b021e 5f66626dee40 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (8m) 40s ago 8m 6405k - 17.2.0 e1d6a67b021e e37d494b0a36 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (7m) 35s ago 8m 52.6M - 8.3.5 dad864ee21e9 ea727583c9f8 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (3m) 35s ago 3m 37.5M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (3m) 40s ago 3m 12.6M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:9283 running (10m) 35s ago 10m 347M - 17.2.0 e1d6a67b021e 8f815fe96db7 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283 running (64s) 40s ago 8m 489M - 19.0.0-1736-g635ee685 c4eb0759a73a 4e3883537bf3 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (10m) 35s ago 10m 50.7M 2048M 17.2.0 e1d6a67b021e 8e3c593dcb9f 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (8m) 40s ago 8m 39.6M 2048M 17.2.0 e1d6a67b021e 976f96599f98 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (2m) 35s ago 2m 252M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:24:05.273 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (2m) 40s ago 2m 73.5M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:24:05.274 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (8m) 35s ago 8m 20.3M - 1.3.1 1dbe0e931976 c20eeeeb0ea2 2024-02-27T06:24:05.274 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (8m) 40s ago 8m 19.7M - 1.3.1 1dbe0e931976 26b7bdf144f9 2024-02-27T06:24:05.274 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (7m) 35s ago 7m 296M 1936M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:24:05.274 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (7m) 35s ago 7m 237M 1936M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:24:05.274 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (6m) 35s ago 6m 187M 1936M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:24:05.274 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (6m) 35s ago 6m 257M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:24:05.274 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (5m) 40s ago 5m 275M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:24:05.274 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (5m) 40s ago 5m 313M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:24:05.274 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (5m) 40s ago 5m 265M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:24:05.274 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (4m) 40s ago 4m 386M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:24:05.274 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (8m) 35s ago 7m 65.3M - 2.33.4 514e6a882f6e 3b3a90b3ef83 2024-02-27T06:24:05.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 96781 13.65 MB/sec execute 43 sec latency 33.641 ms 2024-02-27T06:24:05.523 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:05 smithi131 ceph-mon[24834]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:05.524 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:05 smithi131 ceph-mon[24834]: pgmap v28: 97 pgs: 97 active+clean; 323 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.1 MiB/s wr, 625 op/s 2024-02-27T06:24:05.852 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:24:05.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:24:05.854 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:24:05.854 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-27T06:24:05.854 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T06:24:05.854 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:24:05.854 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:24:05.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:05 smithi150 ceph-mon[26806]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:05.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:05 smithi150 ceph-mon[26806]: pgmap v28: 97 pgs: 97 active+clean; 323 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.1 MiB/s wr, 625 op/s 2024-02-27T06:24:06.412 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:24:06.412 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:24:06.412 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:24:06.412 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:24:06.412 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [], 2024-02-27T06:24:06.412 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "1/23 daemons upgraded", 2024-02-27T06:24:06.412 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "", 2024-02-27T06:24:06.412 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:24:06.413 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:24:06.501 INFO:teuthology.orchestra.run.smithi131.stdout: 5 97236 13.66 MB/sec execute 44 sec latency 37.356 ms 2024-02-27T06:24:06.532 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:06 smithi131 ceph-mon[24834]: from='client.24481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:06.533 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:06 smithi131 ceph-mon[24834]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:06.533 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:06 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1323798328' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:24:06.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:06 smithi150 ceph-mon[26806]: from='client.24481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:06.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:06 smithi150 ceph-mon[26806]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:06.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:06 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1323798328' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:24:07.093 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:24:07.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 97838 13.49 MB/sec execute 45 sec latency 31.268 ms 2024-02-27T06:24:07.541 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:07 smithi131 ceph-mon[24834]: pgmap v29: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 892 op/s 2024-02-27T06:24:07.541 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:07 smithi131 ceph-mon[24834]: from='client.24489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:07.541 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:07 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2417033483' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:24:07.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:07 smithi150 ceph-mon[26806]: pgmap v29: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 892 op/s 2024-02-27T06:24:07.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:07 smithi150 ceph-mon[26806]: from='client.24489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:07.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:07 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/2417033483' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:24:08.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 98269 13.55 MB/sec execute 46 sec latency 33.359 ms 2024-02-27T06:24:09.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 98786 13.61 MB/sec execute 47 sec latency 33.492 ms 2024-02-27T06:24:09.567 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:09 smithi131 ceph-mon[24834]: pgmap v30: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.3 MiB/s wr, 598 op/s 2024-02-27T06:24:09.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:09 smithi150 ceph-mon[26806]: pgmap v30: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.3 MiB/s wr, 598 op/s 2024-02-27T06:24:10.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 99198 13.67 MB/sec execute 48 sec latency 72.920 ms 2024-02-27T06:24:11.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 99514 13.65 MB/sec execute 49 sec latency 37.348 ms 2024-02-27T06:24:11.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:11 smithi131 ceph-mon[24834]: pgmap v31: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.3 MiB/s wr, 598 op/s 2024-02-27T06:24:11.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:11 smithi150 ceph-mon[26806]: pgmap v31: 97 pgs: 97 active+clean; 334 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.3 MiB/s wr, 598 op/s 2024-02-27T06:24:12.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 100035 13.53 MB/sec execute 50 sec latency 26.813 ms 2024-02-27T06:24:13.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:12 smithi150 ceph-mon[26806]: pgmap v32: 97 pgs: 97 active+clean; 328 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 877 op/s 2024-02-27T06:24:13.114 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:12 smithi131 ceph-mon[24834]: pgmap v32: 97 pgs: 97 active+clean; 328 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 877 op/s 2024-02-27T06:24:13.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 100407 13.45 MB/sec execute 51 sec latency 26.629 ms 2024-02-27T06:24:14.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 100769 13.34 MB/sec execute 52 sec latency 81.412 ms 2024-02-27T06:24:15.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 101202 13.21 MB/sec execute 53 sec latency 62.032 ms 2024-02-27T06:24:15.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:15 smithi131 ceph-mon[24834]: pgmap v33: 97 pgs: 97 active+clean; 328 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 546 op/s 2024-02-27T06:24:15.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:15 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:24:15.809 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:15 smithi150 ceph-mon[26806]: pgmap v33: 97 pgs: 97 active+clean; 328 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 546 op/s 2024-02-27T06:24:15.810 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:15 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:24:16.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 101508 13.07 MB/sec execute 54 sec latency 90.721 ms 2024-02-27T06:24:17.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 101877 13.15 MB/sec execute 55 sec latency 73.603 ms 2024-02-27T06:24:17.568 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:17 smithi131 ceph-mon[24834]: pgmap v34: 97 pgs: 97 active+clean; 322 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 799 op/s 2024-02-27T06:24:17.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:17 smithi150 ceph-mon[26806]: pgmap v34: 97 pgs: 97 active+clean; 322 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 799 op/s 2024-02-27T06:24:18.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 102312 13.13 MB/sec execute 56 sec latency 89.790 ms 2024-02-27T06:24:19.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 102717 13.18 MB/sec execute 57 sec latency 33.247 ms 2024-02-27T06:24:19.633 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:19 smithi131 ceph-mon[24834]: pgmap v35: 97 pgs: 97 active+clean; 322 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.9 MiB/s wr, 531 op/s 2024-02-27T06:24:19.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:19 smithi150 ceph-mon[26806]: pgmap v35: 97 pgs: 97 active+clean; 322 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.9 MiB/s wr, 531 op/s 2024-02-27T06:24:20.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 102993 13.13 MB/sec execute 58 sec latency 45.890 ms 2024-02-27T06:24:21.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:21 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:21.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:21 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:21.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:21 smithi131 ceph-mon[24834]: Reconfiguring node-exporter.smithi131 (dependencies changed)... 2024-02-27T06:24:21.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:21 smithi131 ceph-mon[24834]: Reconfiguring daemon node-exporter.smithi131 on smithi131 2024-02-27T06:24:21.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:21 smithi131 ceph-mon[24834]: pgmap v36: 97 pgs: 97 active+clean; 322 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.9 MiB/s wr, 531 op/s 2024-02-27T06:24:21.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:21 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:21.356 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:21 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:21.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:21 smithi150 ceph-mon[26806]: Reconfiguring node-exporter.smithi131 (dependencies changed)... 2024-02-27T06:24:21.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:21 smithi150 ceph-mon[26806]: Reconfiguring daemon node-exporter.smithi131 on smithi131 2024-02-27T06:24:21.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:21 smithi150 ceph-mon[26806]: pgmap v36: 97 pgs: 97 active+clean; 322 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 4.9 MiB/s wr, 531 op/s 2024-02-27T06:24:21.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 103401 13.01 MB/sec execute 59 sec latency 30.497 ms 2024-02-27T06:24:22.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 103737 12.94 MB/sec execute 60 sec latency 40.254 ms 2024-02-27T06:24:23.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:22 smithi150 ceph-mon[26806]: pgmap v37: 97 pgs: 97 active+clean; 330 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 745 op/s 2024-02-27T06:24:23.200 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:22 smithi131 ceph-mon[24834]: pgmap v37: 97 pgs: 97 active+clean; 330 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 745 op/s 2024-02-27T06:24:23.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 104143 12.93 MB/sec execute 61 sec latency 50.376 ms 2024-02-27T06:24:24.472 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:24 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:24.472 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:24 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:24.472 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:24 smithi131 ceph-mon[24834]: Reconfiguring prometheus.smithi131 (dependencies changed)... 2024-02-27T06:24:24.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 104622 12.81 MB/sec execute 62 sec latency 47.621 ms 2024-02-27T06:24:24.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:24 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:24.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:24 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:24.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:24 smithi150 ceph-mon[26806]: Reconfiguring prometheus.smithi131 (dependencies changed)... 2024-02-27T06:24:25.487 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:25 smithi131 ceph-mon[24834]: Reconfiguring daemon prometheus.smithi131 on smithi131 2024-02-27T06:24:25.487 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:25 smithi131 ceph-mon[24834]: pgmap v38: 97 pgs: 97 active+clean; 330 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.8 MiB/s wr, 466 op/s 2024-02-27T06:24:25.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 105075 12.72 MB/sec execute 63 sec latency 32.666 ms 2024-02-27T06:24:25.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:25 smithi150 ceph-mon[26806]: Reconfiguring daemon prometheus.smithi131 on smithi131 2024-02-27T06:24:25.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:25 smithi150 ceph-mon[26806]: pgmap v38: 97 pgs: 97 active+clean; 330 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.8 MiB/s wr, 466 op/s 2024-02-27T06:24:26.502 INFO:teuthology.orchestra.run.smithi131.stdout: 5 105600 12.85 MB/sec execute 64 sec latency 33.893 ms 2024-02-27T06:24:27.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 106154 12.95 MB/sec execute 65 sec latency 32.836 ms 2024-02-27T06:24:27.510 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:27 smithi131 ceph-mon[24834]: pgmap v39: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 725 op/s 2024-02-27T06:24:27.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:27 smithi150 ceph-mon[26806]: pgmap v39: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 725 op/s 2024-02-27T06:24:28.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 106440 12.93 MB/sec execute 66 sec latency 47.630 ms 2024-02-27T06:24:29.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 106753 12.85 MB/sec execute 67 sec latency 34.563 ms 2024-02-27T06:24:29.544 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:29 smithi131 ceph-mon[24834]: pgmap v40: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.9 MiB/s wr, 472 op/s 2024-02-27T06:24:29.544 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:29 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:24:29.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:29 smithi150 ceph-mon[26806]: pgmap v40: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.9 MiB/s wr, 472 op/s 2024-02-27T06:24:29.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:29 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:24:30.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 107087 12.76 MB/sec execute 68 sec latency 42.510 ms 2024-02-27T06:24:31.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 107650 12.81 MB/sec execute 69 sec latency 25.071 ms 2024-02-27T06:24:31.567 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:31 smithi131 ceph-mon[24834]: pgmap v41: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.9 MiB/s wr, 472 op/s 2024-02-27T06:24:31.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:31 smithi150 ceph-mon[26806]: pgmap v41: 97 pgs: 97 active+clean; 324 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.9 MiB/s wr, 472 op/s 2024-02-27T06:24:32.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 108201 12.71 MB/sec execute 70 sec latency 41.831 ms 2024-02-27T06:24:33.074 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:32 smithi131 ceph-mon[24834]: pgmap v42: 97 pgs: 97 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.6 MiB/s wr, 708 op/s 2024-02-27T06:24:33.106 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:32 smithi150 ceph-mon[26806]: pgmap v42: 97 pgs: 97 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.6 MiB/s wr, 708 op/s 2024-02-27T06:24:33.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 109004 12.89 MB/sec execute 71 sec latency 22.770 ms 2024-02-27T06:24:34.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 109843 13.10 MB/sec execute 72 sec latency 25.571 ms 2024-02-27T06:24:35.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 110319 13.10 MB/sec execute 73 sec latency 43.155 ms 2024-02-27T06:24:35.601 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:35 smithi131 ceph-mon[24834]: pgmap v43: 97 pgs: 97 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.1 MiB/s wr, 494 op/s 2024-02-27T06:24:35.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:35 smithi150 ceph-mon[26806]: pgmap v43: 97 pgs: 97 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.1 MiB/s wr, 494 op/s 2024-02-27T06:24:36.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 110745 13.04 MB/sec execute 74 sec latency 37.154 ms 2024-02-27T06:24:37.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 111077 13.00 MB/sec execute 75 sec latency 48.559 ms 2024-02-27T06:24:37.562 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:24:37.631 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:37 smithi131 ceph-mon[24834]: pgmap v44: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 877 op/s 2024-02-27T06:24:37.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:37 smithi150 ceph-mon[26806]: pgmap v44: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 877 op/s 2024-02-27T06:24:38.460 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:24:38.460 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 starting - - - - 2024-02-27T06:24:38.461 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (9m) 68s ago 9m 6405k - 17.2.0 e1d6a67b021e 5f66626dee40 2024-02-27T06:24:38.461 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (9m) 73s ago 9m 6405k - 17.2.0 e1d6a67b021e e37d494b0a36 2024-02-27T06:24:38.461 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 starting - - - - 2024-02-27T06:24:38.461 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (3m) 68s ago 3m 37.5M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:24:38.461 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (3m) 73s ago 3m 12.6M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:24:38.461 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:9283 running (10m) 68s ago 10m 347M - 17.2.0 e1d6a67b021e 8f815fe96db7 2024-02-27T06:24:38.461 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283 running (97s) 73s ago 9m 489M - 19.0.0-1736-g635ee685 c4eb0759a73a 4e3883537bf3 2024-02-27T06:24:38.461 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (10m) 68s ago 10m 50.7M 2048M 17.2.0 e1d6a67b021e 8e3c593dcb9f 2024-02-27T06:24:38.461 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (8m) 73s ago 8m 39.6M 2048M 17.2.0 e1d6a67b021e 976f96599f98 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (3m) 68s ago 3m 252M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (3m) 73s ago 3m 73.5M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 starting - - - - 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (8m) 73s ago 8m 19.7M - 1.3.1 1dbe0e931976 26b7bdf144f9 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (8m) 68s ago 8m 296M 1936M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (7m) 68s ago 7m 237M 1936M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (7m) 68s ago 7m 187M 1936M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (6m) 68s ago 6m 257M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (6m) 73s ago 6m 275M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (6m) 73s ago 6m 313M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (5m) 73s ago 5m 265M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (5m) 73s ago 5m 386M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:24:38.462 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (8m) 68s ago 8m 65.3M - 2.33.4 514e6a882f6e 3b3a90b3ef83 2024-02-27T06:24:38.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 111539 12.91 MB/sec execute 76 sec latency 46.876 ms 2024-02-27T06:24:38.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:38 smithi131 ceph-mon[24834]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:38.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:38 smithi150 ceph-mon[26806]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:39.005 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:24:39.005 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:24:39.005 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T06:24:39.006 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:24:39.007 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:24:39.494 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:24:39.494 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:24:39.495 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:24:39.495 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:24:39.495 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [], 2024-02-27T06:24:39.495 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "1/23 daemons upgraded", 2024-02-27T06:24:39.495 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "", 2024-02-27T06:24:39.495 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:24:39.495 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:24:39.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 111945 12.85 MB/sec execute 77 sec latency 37.393 ms 2024-02-27T06:24:39.600 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:39 smithi131 ceph-mon[24834]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:39.600 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:39 smithi131 ceph-mon[24834]: pgmap v45: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.8 MiB/s wr, 617 op/s 2024-02-27T06:24:39.600 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:39 smithi131 ceph-mon[24834]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:39.600 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:39 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2336622561' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:24:39.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:39 smithi150 ceph-mon[26806]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:39.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:39 smithi150 ceph-mon[26806]: pgmap v45: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.8 MiB/s wr, 617 op/s 2024-02-27T06:24:39.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:39 smithi150 ceph-mon[26806]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:39.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:39 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/2336622561' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:24:40.064 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:24:40.503 INFO:teuthology.orchestra.run.smithi131.stdout: 5 112354 12.85 MB/sec execute 78 sec latency 59.890 ms 2024-02-27T06:24:40.615 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:40 smithi131 ceph-mon[24834]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:40.615 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:40 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/1155795651' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:24:40.856 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:40 smithi150 ceph-mon[26806]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:24:40.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:40 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/1155795651' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:24:41.508 INFO:teuthology.orchestra.run.smithi131.stdout: 5 112753 12.87 MB/sec execute 79 sec latency 57.685 ms 2024-02-27T06:24:41.627 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:41 smithi131 ceph-mon[24834]: pgmap v46: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.8 MiB/s wr, 617 op/s 2024-02-27T06:24:41.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:41 smithi150 ceph-mon[26806]: pgmap v46: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 6.8 MiB/s wr, 617 op/s 2024-02-27T06:24:42.508 INFO:teuthology.orchestra.run.smithi131.stdout: 5 113280 12.95 MB/sec execute 80 sec latency 30.114 ms 2024-02-27T06:24:43.107 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:42 smithi150 ceph-mon[26806]: pgmap v47: 97 pgs: 97 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 868 op/s 2024-02-27T06:24:43.143 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:42 smithi131 ceph-mon[24834]: pgmap v47: 97 pgs: 97 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.9 MiB/s wr, 868 op/s 2024-02-27T06:24:43.508 INFO:teuthology.orchestra.run.smithi131.stdout: 5 113641 12.94 MB/sec execute 81 sec latency 37.853 ms 2024-02-27T06:24:44.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 114011 12.85 MB/sec execute 82 sec latency 34.707 ms 2024-02-27T06:24:45.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 114332 12.79 MB/sec execute 83 sec latency 52.870 ms 2024-02-27T06:24:45.689 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:45 smithi131 ceph-mon[24834]: pgmap v48: 97 pgs: 97 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.2 MiB/s wr, 633 op/s 2024-02-27T06:24:45.689 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:45 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:24:45.810 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:45 smithi150 ceph-mon[26806]: pgmap v48: 97 pgs: 97 active+clean; 328 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.2 MiB/s wr, 633 op/s 2024-02-27T06:24:45.810 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:45 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:24:46.491 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:46 smithi131 ceph-mon[24834]: mgrmap e27: smithi150.vvgbyr(active, since 92s), standbys: smithi131.frtgqo 2024-02-27T06:24:46.491 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:46 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:46.491 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:46 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:46.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 114681 12.75 MB/sec execute 84 sec latency 48.902 ms 2024-02-27T06:24:46.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:46 smithi150 ceph-mon[26806]: mgrmap e27: smithi150.vvgbyr(active, since 92s), standbys: smithi131.frtgqo 2024-02-27T06:24:46.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:46 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:46.727 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:46 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:47.504 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:47 smithi131 ceph-mon[24834]: Reconfiguring node-exporter.smithi150 (dependencies changed)... 2024-02-27T06:24:47.504 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:47 smithi131 ceph-mon[24834]: Reconfiguring daemon node-exporter.smithi150 on smithi150 2024-02-27T06:24:47.504 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:47 smithi131 ceph-mon[24834]: pgmap v49: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 873 op/s 2024-02-27T06:24:47.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 115178 12.69 MB/sec execute 85 sec latency 42.982 ms 2024-02-27T06:24:47.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:47 smithi150 ceph-mon[26806]: Reconfiguring node-exporter.smithi150 (dependencies changed)... 2024-02-27T06:24:47.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:47 smithi150 ceph-mon[26806]: Reconfiguring daemon node-exporter.smithi150 on smithi150 2024-02-27T06:24:47.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:47 smithi150 ceph-mon[26806]: pgmap v49: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 873 op/s 2024-02-27T06:24:48.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 115526 12.63 MB/sec execute 86 sec latency 44.540 ms 2024-02-27T06:24:49.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:49.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:49.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T06:24:49.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T06:24:49.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi131.front.sepia.ceph.com:9093"}]: dispatch 2024-02-27T06:24:49.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi131.front.sepia.ceph.com:9093"}]: dispatch 2024-02-27T06:24:49.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:49.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T06:24:49.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T06:24:49.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi131.front.sepia.ceph.com:3000"}]: dispatch 2024-02-27T06:24:49.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi131.front.sepia.ceph.com:3000"}]: dispatch 2024-02-27T06:24:49.303 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:49.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:24:49.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:24:49.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi131.front.sepia.ceph.com:9095"}]: dispatch 2024-02-27T06:24:49.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi131.front.sepia.ceph.com:9095"}]: dispatch 2024-02-27T06:24:49.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:49.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:24:49.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: Upgrade: Need to upgrade myself (mgr.smithi150.vvgbyr) 2024-02-27T06:24:49.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: Upgrade: Need to upgrade myself (mgr.smithi150.vvgbyr) 2024-02-27T06:24:49.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:49 smithi131 ceph-mon[24834]: pgmap v50: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.0 MiB/s wr, 491 op/s 2024-02-27T06:24:49.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 115953 12.64 MB/sec execute 87 sec latency 38.586 ms 2024-02-27T06:24:49.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:49.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:49.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T06:24:49.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T06:24:49.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi131.front.sepia.ceph.com:9093"}]: dispatch 2024-02-27T06:24:49.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi131.front.sepia.ceph.com:9093"}]: dispatch 2024-02-27T06:24:49.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:49.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T06:24:49.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T06:24:49.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi131.front.sepia.ceph.com:3000"}]: dispatch 2024-02-27T06:24:49.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi131.front.sepia.ceph.com:3000"}]: dispatch 2024-02-27T06:24:49.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:49.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:24:49.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:24:49.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi131.front.sepia.ceph.com:9095"}]: dispatch 2024-02-27T06:24:49.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi131.front.sepia.ceph.com:9095"}]: dispatch 2024-02-27T06:24:49.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:49.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:24:49.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: Upgrade: Need to upgrade myself (mgr.smithi150.vvgbyr) 2024-02-27T06:24:49.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: Upgrade: Need to upgrade myself (mgr.smithi150.vvgbyr) 2024-02-27T06:24:49.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:49 smithi150 ceph-mon[26806]: pgmap v50: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.0 MiB/s wr, 491 op/s 2024-02-27T06:24:50.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 116437 12.69 MB/sec execute 88 sec latency 44.932 ms 2024-02-27T06:24:51.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 116986 12.76 MB/sec execute 89 sec latency 30.526 ms 2024-02-27T06:24:51.744 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:51 smithi131 ceph-mon[24834]: pgmap v51: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.0 MiB/s wr, 491 op/s 2024-02-27T06:24:51.744 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:51 smithi131 ceph-mon[24834]: Upgrade: Updating mgr.smithi131.frtgqo 2024-02-27T06:24:51.744 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:51 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:51.744 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:51 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi131.frtgqo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:24:51.744 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:51 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi131.frtgqo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:24:51.744 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:51 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:24:51.744 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:51 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:24:51.744 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:51 smithi131 ceph-mon[24834]: Deploying daemon mgr.smithi131.frtgqo on smithi131 2024-02-27T06:24:52.107 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:51 smithi150 ceph-mon[26806]: pgmap v51: 97 pgs: 97 active+clean; 335 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.0 MiB/s wr, 491 op/s 2024-02-27T06:24:52.107 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:51 smithi150 ceph-mon[26806]: Upgrade: Updating mgr.smithi131.frtgqo 2024-02-27T06:24:52.107 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:51 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:24:52.107 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:51 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi131.frtgqo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:24:52.107 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:51 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi131.frtgqo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:24:52.107 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:51 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:24:52.107 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:51 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:24:52.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:51 smithi150 ceph-mon[26806]: Deploying daemon mgr.smithi131.frtgqo on smithi131 2024-02-27T06:24:52.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 117398 12.75 MB/sec execute 90 sec latency 48.019 ms 2024-02-27T06:24:53.123 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:52 smithi131 ceph-mon[24834]: pgmap v52: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.3 MiB/s wr, 742 op/s 2024-02-27T06:24:53.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:52 smithi150 ceph-mon[26806]: pgmap v52: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 7.3 MiB/s wr, 742 op/s 2024-02-27T06:24:53.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 117792 12.70 MB/sec execute 91 sec latency 103.204 ms 2024-02-27T06:24:54.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 118161 12.68 MB/sec execute 92 sec latency 160.901 ms 2024-02-27T06:24:55.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 118599 12.61 MB/sec execute 93 sec latency 52.534 ms 2024-02-27T06:24:55.637 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:55 smithi131 ceph-mon[24834]: pgmap v53: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.1 MiB/s wr, 491 op/s 2024-02-27T06:24:55.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:55 smithi150 ceph-mon[26806]: pgmap v53: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.1 MiB/s wr, 491 op/s 2024-02-27T06:24:56.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 119000 12.55 MB/sec execute 94 sec latency 53.012 ms 2024-02-27T06:24:57.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:57 smithi131 ceph-mon[24834]: pgmap v54: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 758 op/s 2024-02-27T06:24:57.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 119410 12.56 MB/sec execute 95 sec latency 31.125 ms 2024-02-27T06:24:57.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:57 smithi150 ceph-mon[26806]: pgmap v54: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 758 op/s 2024-02-27T06:24:58.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 120080 12.64 MB/sec execute 96 sec latency 31.189 ms 2024-02-27T06:24:59.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:24:59 smithi131 ceph-mon[24834]: pgmap v55: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 517 op/s 2024-02-27T06:24:59.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 120875 12.80 MB/sec execute 97 sec latency 19.729 ms 2024-02-27T06:24:59.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:24:59 smithi150 ceph-mon[26806]: pgmap v55: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 517 op/s 2024-02-27T06:25:00.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:00 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:25:00.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:00 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:00.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:00 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:00.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:00 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:25:00.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:00 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:00.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:00 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:00.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 121329 12.77 MB/sec execute 98 sec latency 54.907 ms 2024-02-27T06:25:00.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:00 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:25:00.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:00 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:00.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:00 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:00.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:00 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:25:00.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:00 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:00.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:00 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:01.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 121848 12.76 MB/sec execute 99 sec latency 19.849 ms 2024-02-27T06:25:01.715 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:01 smithi131 ceph-mon[24834]: pgmap v56: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 517 op/s 2024-02-27T06:25:01.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:01 smithi150 ceph-mon[26806]: pgmap v56: 97 pgs: 97 active+clean; 331 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.0 MiB/s wr, 517 op/s 2024-02-27T06:25:02.509 INFO:teuthology.orchestra.run.smithi131.stdout: 5 122317 12.70 MB/sec execute 100 sec latency 39.392 ms 2024-02-27T06:25:02.995 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:02 smithi131 ceph-mon[24834]: pgmap v57: 97 pgs: 97 active+clean; 333 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 842 op/s 2024-02-27T06:25:03.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:02 smithi150 ceph-mon[26806]: pgmap v57: 97 pgs: 97 active+clean; 333 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 842 op/s 2024-02-27T06:25:03.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 122716 12.69 MB/sec execute 101 sec latency 107.215 ms 2024-02-27T06:25:04.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 123206 12.74 MB/sec execute 102 sec latency 51.977 ms 2024-02-27T06:25:05.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 123756 12.79 MB/sec execute 103 sec latency 85.269 ms 2024-02-27T06:25:05.516 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:05 smithi131 ceph-mon[24834]: pgmap v58: 97 pgs: 97 active+clean; 333 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.3 MiB/s wr, 591 op/s 2024-02-27T06:25:05.516 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:05 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:05.516 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:05 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:05.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:05 smithi150 ceph-mon[26806]: pgmap v58: 97 pgs: 97 active+clean; 333 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 6.3 MiB/s wr, 591 op/s 2024-02-27T06:25:05.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:05 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:05.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:05 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:06.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 124112 12.77 MB/sec execute 104 sec latency 37.178 ms 2024-02-27T06:25:07.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 124521 12.75 MB/sec execute 105 sec latency 54.843 ms 2024-02-27T06:25:07.532 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:07 smithi131 ceph-mon[24834]: pgmap v59: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 872 op/s 2024-02-27T06:25:07.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:07 smithi150 ceph-mon[26806]: pgmap v59: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 872 op/s 2024-02-27T06:25:08.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 124858 12.71 MB/sec execute 106 sec latency 42.000 ms 2024-02-27T06:25:09.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 125187 12.66 MB/sec execute 107 sec latency 41.549 ms 2024-02-27T06:25:09.572 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:09 smithi131 ceph-mon[24834]: pgmap v60: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 605 op/s 2024-02-27T06:25:09.572 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:09 smithi131 ceph-mon[24834]: Standby manager daemon smithi131.frtgqo restarted 2024-02-27T06:25:09.572 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:09 smithi131 ceph-mon[24834]: Standby manager daemon smithi131.frtgqo started 2024-02-27T06:25:09.572 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:09 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi131.frtgqo/crt"}]: dispatch 2024-02-27T06:25:09.572 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:09 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:25:09.573 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:09 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi131.frtgqo/key"}]: dispatch 2024-02-27T06:25:09.573 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:09 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:25:09.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:09 smithi150 ceph-mon[26806]: pgmap v60: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 605 op/s 2024-02-27T06:25:09.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:09 smithi150 ceph-mon[26806]: Standby manager daemon smithi131.frtgqo restarted 2024-02-27T06:25:09.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:09 smithi150 ceph-mon[26806]: Standby manager daemon smithi131.frtgqo started 2024-02-27T06:25:09.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:09 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi131.frtgqo/crt"}]: dispatch 2024-02-27T06:25:09.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:09 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:25:09.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:09 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi131.frtgqo/key"}]: dispatch 2024-02-27T06:25:09.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:09 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:25:10.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 125791 12.64 MB/sec execute 108 sec latency 33.584 ms 2024-02-27T06:25:10.578 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:10 smithi131 ceph-mon[24834]: mgrmap e28: smithi150.vvgbyr(active, since 116s), standbys: smithi131.frtgqo 2024-02-27T06:25:10.661 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:25:10.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:10 smithi150 ceph-mon[26806]: mgrmap e28: smithi150.vvgbyr(active, since 116s), standbys: smithi131.frtgqo 2024-02-27T06:25:11.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 126399 12.66 MB/sec execute 109 sec latency 32.543 ms 2024-02-27T06:25:11.585 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:11 smithi131 ceph-mon[24834]: pgmap v61: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 605 op/s 2024-02-27T06:25:11.585 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:11 smithi131 ceph-mon[24834]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:11.585 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:11 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:11.585 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:11 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:11.706 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:25:11.706 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (81s) 0s ago 10m 23.4M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:25:11.706 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (10m) 0s ago 10m 6405k - 17.2.0 e1d6a67b021e 5f66626dee40 2024-02-27T06:25:11.706 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (9m) 6s ago 9m 6405k - 17.2.0 e1d6a67b021e e37d494b0a36 2024-02-27T06:25:11.706 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (52s) 0s ago 9m 46.3M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:25:11.706 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (4m) 0s ago 4m 38.4M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:25:11.706 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (4m) 6s ago 4m 13.1M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:25:11.706 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (13s) 0s ago 11m 95.8M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283 running (2m) 6s ago 9m 499M - 19.0.0-1736-g635ee685 c4eb0759a73a 4e3883537bf3 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (11m) 0s ago 11m 52.7M 2048M 17.2.0 e1d6a67b021e 8e3c593dcb9f 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (9m) 6s ago 9m 39.0M 2048M 17.2.0 e1d6a67b021e 976f96599f98 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (4m) 0s ago 4m 253M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (3m) 6s ago 3m 79.3M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (49s) 0s ago 9m 10.2M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (23s) 6s ago 9m 8975k - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (8m) 0s ago 8m 445M 1936M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (8m) 0s ago 8m 400M 1936M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (7m) 0s ago 7m 276M 1936M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (7m) 0s ago 7m 412M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (7m) 6s ago 7m 403M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (6m) 6s ago 6m 454M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (6m) 6s ago 6m 403M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:25:11.707 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (5m) 6s ago 5m 526M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:25:11.708 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (26s) 0s ago 9m 48.7M - 2.33.4 514e6a882f6e 17b59b7362c7 2024-02-27T06:25:11.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:11 smithi150 ceph-mon[26806]: pgmap v61: 97 pgs: 97 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 605 op/s 2024-02-27T06:25:11.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:11 smithi150 ceph-mon[26806]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:11.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:11 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:11.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:11 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:12.251 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:25:12.252 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:25:12.253 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-27T06:25:12.253 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:25:12.253 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:25:12.253 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:25:12.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 126889 12.72 MB/sec execute 110 sec latency 69.220 ms 2024-02-27T06:25:12.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:12 smithi131 ceph-mon[24834]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:12.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:12 smithi131 ceph-mon[24834]: from='client.24519 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:12.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:12 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/2092667677' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:25:12.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:12 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:12.594 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:12 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:12.703 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:25:12.704 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:25:12.704 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:25:12.704 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:25:12.704 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:25:12.704 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr" 2024-02-27T06:25:12.704 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:25:12.704 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "2/23 daemons upgraded", 2024-02-27T06:25:12.704 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading mgr daemons", 2024-02-27T06:25:12.704 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:25:12.704 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:25:12.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:12 smithi150 ceph-mon[26806]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:12.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:12 smithi150 ceph-mon[26806]: from='client.24519 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:12.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:12 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/2092667677' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:25:12.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:12 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:12.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:12 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:13.250 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:25:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 127280 12.69 MB/sec execute 111 sec latency 37.659 ms 2024-02-27T06:25:13.603 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:13 smithi131 ceph-mon[24834]: pgmap v62: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 859 op/s 2024-02-27T06:25:13.603 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:13 smithi131 ceph-mon[24834]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:13.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:13 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/278356259' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:25:13.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:13 smithi150 ceph-mon[26806]: pgmap v62: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.8 MiB/s wr, 859 op/s 2024-02-27T06:25:13.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:13 smithi150 ceph-mon[26806]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:13.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:13 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/278356259' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:25:14.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 127610 12.69 MB/sec execute 112 sec latency 38.257 ms 2024-02-27T06:25:15.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 127925 12.68 MB/sec execute 113 sec latency 33.998 ms 2024-02-27T06:25:15.635 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:15 smithi131 ceph-mon[24834]: pgmap v63: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 534 op/s 2024-02-27T06:25:15.636 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:15 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:25:15.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:15 smithi150 ceph-mon[26806]: pgmap v63: 97 pgs: 97 active+clean; 328 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 534 op/s 2024-02-27T06:25:15.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:15 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:25:16.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 128187 12.61 MB/sec execute 114 sec latency 57.852 ms 2024-02-27T06:25:17.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 128448 12.55 MB/sec execute 115 sec latency 46.783 ms 2024-02-27T06:25:17.703 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:17 smithi131 ceph-mon[24834]: pgmap v64: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 790 op/s 2024-02-27T06:25:17.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:17 smithi150 ceph-mon[26806]: pgmap v64: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 790 op/s 2024-02-27T06:25:18.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 128750 12.49 MB/sec execute 116 sec latency 52.861 ms 2024-02-27T06:25:19.215 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:19.215 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:19.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:25:19.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:25:19.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:19.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:19.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:25:19.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: Upgrade: Need to upgrade myself (mgr.smithi150.vvgbyr) 2024-02-27T06:25:19.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: Upgrade: Need to upgrade myself (mgr.smithi150.vvgbyr) 2024-02-27T06:25:19.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: Failing over to other MGR 2024-02-27T06:25:19.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mgr fail", "who": "smithi150.vvgbyr"}]: dispatch 2024-02-27T06:25:19.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mgr fail", "who": "smithi150.vvgbyr"}]: dispatch 2024-02-27T06:25:19.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: osdmap e72: 8 total, 8 up, 8 in 2024-02-27T06:25:19.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:19 smithi131 ceph-mon[24834]: pgmap v66: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.4 MiB/s wr, 612 op/s 2024-02-27T06:25:19.323 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:19.323 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:19.323 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:25:19.323 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:25:19.324 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:19.324 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' 2024-02-27T06:25:19.324 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:25:19.324 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: Upgrade: Need to upgrade myself (mgr.smithi150.vvgbyr) 2024-02-27T06:25:19.324 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: Upgrade: Need to upgrade myself (mgr.smithi150.vvgbyr) 2024-02-27T06:25:19.324 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: Failing over to other MGR 2024-02-27T06:25:19.325 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: from='mgr.24425 172.21.15.150:0/1942195848' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mgr fail", "who": "smithi150.vvgbyr"}]: dispatch 2024-02-27T06:25:19.325 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "mgr fail", "who": "smithi150.vvgbyr"}]: dispatch 2024-02-27T06:25:19.325 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: osdmap e72: 8 total, 8 up, 8 in 2024-02-27T06:25:19.325 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:19 smithi150 ceph-mon[26806]: pgmap v66: 97 pgs: 97 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.4 MiB/s wr, 612 op/s 2024-02-27T06:25:19.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 129077 12.44 MB/sec execute 117 sec latency 75.955 ms 2024-02-27T06:25:20.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd='[{"prefix": "mgr fail", "who": "smithi150.vvgbyr"}]': finished 2024-02-27T06:25:20.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: mgrmap e29: smithi131.frtgqo(active, starting, since 0.858074s) 2024-02-27T06:25:20.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:25:20.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:25:20.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:25:20.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:25:20.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi131.frtgqo", "id": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:25:20.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: Manager daemon smithi131.frtgqo is now available 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:25:20.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:25:20.282 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:25:20.282 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:20 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:25:20.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24425 ' entity='mgr.smithi150.vvgbyr' cmd='[{"prefix": "mgr fail", "who": "smithi150.vvgbyr"}]': finished 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: mgrmap e29: smithi131.frtgqo(active, starting, since 0.858074s) 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi131.frtgqo", "id": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:25:20.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:25:20.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:25:20.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T06:25:20.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T06:25:20.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T06:25:20.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: Manager daemon smithi131.frtgqo is now available 2024-02-27T06:25:20.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:25:20.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:25:20.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:25:20.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:25:20.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:20 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:25:20.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 129323 12.38 MB/sec execute 118 sec latency 73.783 ms 2024-02-27T06:25:21.265 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:21 smithi150 ceph-mon[26806]: [27/Feb/2024:06:25:19] ENGINE Bus STARTING 2024-02-27T06:25:21.265 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:21 smithi150 ceph-mon[26806]: [27/Feb/2024:06:25:19] ENGINE Serving on http://172.21.15.131:8765 2024-02-27T06:25:21.265 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:21 smithi150 ceph-mon[26806]: [27/Feb/2024:06:25:19] ENGINE Serving on https://172.21.15.131:7150 2024-02-27T06:25:21.265 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:21 smithi150 ceph-mon[26806]: [27/Feb/2024:06:25:19] ENGINE Bus STARTED 2024-02-27T06:25:21.265 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:21 smithi150 ceph-mon[26806]: mgrmap e30: smithi131.frtgqo(active, since 1.86731s) 2024-02-27T06:25:21.265 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:21 smithi150 ceph-mon[26806]: pgmap v3: 97 pgs: 97 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:25:21.301 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:21 smithi131 ceph-mon[24834]: [27/Feb/2024:06:25:19] ENGINE Bus STARTING 2024-02-27T06:25:21.301 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:21 smithi131 ceph-mon[24834]: [27/Feb/2024:06:25:19] ENGINE Serving on http://172.21.15.131:8765 2024-02-27T06:25:21.301 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:21 smithi131 ceph-mon[24834]: [27/Feb/2024:06:25:19] ENGINE Serving on https://172.21.15.131:7150 2024-02-27T06:25:21.301 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:21 smithi131 ceph-mon[24834]: [27/Feb/2024:06:25:19] ENGINE Bus STARTED 2024-02-27T06:25:21.301 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:21 smithi131 ceph-mon[24834]: mgrmap e30: smithi131.frtgqo(active, since 1.86731s) 2024-02-27T06:25:21.301 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:21 smithi131 ceph-mon[24834]: pgmap v3: 97 pgs: 97 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:25:21.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 129610 12.33 MB/sec execute 119 sec latency 47.500 ms 2024-02-27T06:25:22.065 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:22 smithi131 ceph-mon[24834]: pgmap v4: 97 pgs: 97 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:25:22.065 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:22 smithi131 ceph-mon[24834]: mgrmap e31: smithi131.frtgqo(active, since 2s) 2024-02-27T06:25:22.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:22 smithi150 ceph-mon[26806]: pgmap v4: 97 pgs: 97 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:25:22.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:22 smithi150 ceph-mon[26806]: mgrmap e31: smithi131.frtgqo(active, since 2s) 2024-02-27T06:25:22.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 130168 12.38 MB/sec execute 120 sec latency 31.345 ms 2024-02-27T06:25:23.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 130729 12.42 MB/sec execute 121 sec latency 30.159 ms 2024-02-27T06:25:24.075 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:24 smithi131 ceph-mon[24834]: pgmap v5: 97 pgs: 97 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:25:24.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:24 smithi131 ceph-mon[24834]: mgrmap e32: smithi131.frtgqo(active, since 4s) 2024-02-27T06:25:24.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:24 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:24.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:24 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:24.324 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:24 smithi150 ceph-mon[26806]: pgmap v5: 97 pgs: 97 active+clean; 329 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:25:24.324 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:24 smithi150 ceph-mon[26806]: mgrmap e32: smithi131.frtgqo(active, since 4s) 2024-02-27T06:25:24.324 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:24 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:24.324 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:24 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:24.510 INFO:teuthology.orchestra.run.smithi131.stdout: 5 131120 12.43 MB/sec execute 122 sec latency 48.469 ms 2024-02-27T06:25:25.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 131417 12.41 MB/sec execute 123 sec latency 58.985 ms 2024-02-27T06:25:26.139 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:26 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:26.139 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:26 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:26.139 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:26 smithi131 ceph-mon[24834]: pgmap v6: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 1.8 MiB/s wr, 62 op/s 2024-02-27T06:25:26.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:26 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:26.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:26 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:26.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:26 smithi150 ceph-mon[26806]: pgmap v6: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 1.8 MiB/s wr, 62 op/s 2024-02-27T06:25:26.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 131662 12.35 MB/sec execute 124 sec latency 58.059 ms 2024-02-27T06:25:27.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 131909 12.30 MB/sec execute 125 sec latency 56.382 ms 2024-02-27T06:25:28.048 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:27 smithi150 ceph-mon[26806]: pgmap v7: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 34 KiB/s rd, 4.6 MiB/s wr, 430 op/s 2024-02-27T06:25:28.205 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:27 smithi131 ceph-mon[24834]: pgmap v7: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 34 KiB/s rd, 4.6 MiB/s wr, 430 op/s 2024-02-27T06:25:28.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 132212 12.25 MB/sec execute 126 sec latency 55.058 ms 2024-02-27T06:25:29.211 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:28 smithi131 ceph-mon[24834]: Standby manager daemon smithi150.vvgbyr started 2024-02-27T06:25:29.211 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:28 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/1204770080' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/crt"}]: dispatch 2024-02-27T06:25:29.211 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:28 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/1204770080' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:25:29.212 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:28 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/1204770080' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/key"}]: dispatch 2024-02-27T06:25:29.212 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:28 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/1204770080' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:25:29.212 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:28 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:29.212 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:28 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:29.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:28 smithi150 ceph-mon[26806]: Standby manager daemon smithi150.vvgbyr started 2024-02-27T06:25:29.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:28 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/1204770080' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/crt"}]: dispatch 2024-02-27T06:25:29.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:28 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/1204770080' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:25:29.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:28 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/1204770080' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/key"}]: dispatch 2024-02-27T06:25:29.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:28 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/1204770080' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:25:29.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:28 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:29.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:28 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:29.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 132594 12.21 MB/sec execute 127 sec latency 43.712 ms 2024-02-27T06:25:30.003 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:29 smithi131 ceph-mon[24834]: mgrmap e33: smithi131.frtgqo(active, since 10s), standbys: smithi150.vvgbyr 2024-02-27T06:25:30.004 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:29 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi150.vvgbyr", "id": "smithi150.vvgbyr"}]: dispatch 2024-02-27T06:25:30.004 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:29 smithi131 ceph-mon[24834]: pgmap v8: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 3.6 MiB/s wr, 334 op/s 2024-02-27T06:25:30.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:29 smithi150 ceph-mon[26806]: mgrmap e33: smithi131.frtgqo(active, since 10s), standbys: smithi150.vvgbyr 2024-02-27T06:25:30.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:29 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi150.vvgbyr", "id": "smithi150.vvgbyr"}]: dispatch 2024-02-27T06:25:30.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:29 smithi150 ceph-mon[26806]: pgmap v8: 97 pgs: 97 active+clean; 328 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 3.6 MiB/s wr, 334 op/s 2024-02-27T06:25:30.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 132946 12.17 MB/sec execute 128 sec latency 56.156 ms 2024-02-27T06:25:31.038 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:31 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:31.039 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:31 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:31.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:31 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:31.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:31 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:31.512 INFO:teuthology.orchestra.run.smithi131.stdout: 5 133462 12.18 MB/sec execute 129 sec latency 44.434 ms 2024-02-27T06:25:32.081 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:32 smithi131 ceph-mon[24834]: pgmap v9: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.8 MiB/s wr, 486 op/s 2024-02-27T06:25:32.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:32 smithi150 ceph-mon[26806]: pgmap v9: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.8 MiB/s wr, 486 op/s 2024-02-27T06:25:32.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 133962 12.21 MB/sec execute 130 sec latency 50.328 ms 2024-02-27T06:25:33.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 134435 12.24 MB/sec execute 131 sec latency 41.319 ms 2024-02-27T06:25:34.175 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:34 smithi131 ceph-mon[24834]: pgmap v10: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.4 MiB/s wr, 444 op/s 2024-02-27T06:25:34.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 134701 12.22 MB/sec execute 132 sec latency 43.926 ms 2024-02-27T06:25:34.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:34 smithi150 ceph-mon[26806]: pgmap v10: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.4 MiB/s wr, 444 op/s 2024-02-27T06:25:35.184 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:35 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:25:35.184 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:35 smithi131 ceph-mon[24834]: pgmap v11: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.1 MiB/s wr, 471 op/s 2024-02-27T06:25:35.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 134989 12.20 MB/sec execute 133 sec latency 71.772 ms 2024-02-27T06:25:35.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:35 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:25:35.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:35 smithi150 ceph-mon[26806]: pgmap v11: 97 pgs: 97 active+clean; 327 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.1 MiB/s wr, 471 op/s 2024-02-27T06:25:36.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 135241 12.14 MB/sec execute 134 sec latency 54.541 ms 2024-02-27T06:25:37.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:37 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:37.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:37 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:37.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:37 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:25:37.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:37 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:25:37.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:37 smithi131 ceph-mon[24834]: Updating smithi131:/etc/ceph/ceph.conf 2024-02-27T06:25:37.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:37 smithi131 ceph-mon[24834]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-27T06:25:37.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:37 smithi131 ceph-mon[24834]: Updating smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T06:25:37.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:37 smithi131 ceph-mon[24834]: Updating smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T06:25:37.265 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:37 smithi131 ceph-mon[24834]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:25:37.266 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:37 smithi131 ceph-mon[24834]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:25:37.266 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:37 smithi131 ceph-mon[24834]: Updating smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T06:25:37.266 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:37 smithi131 ceph-mon[24834]: pgmap v12: 97 pgs: 97 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 6.3 MiB/s wr, 662 op/s 2024-02-27T06:25:37.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 135567 12.13 MB/sec execute 135 sec latency 39.791 ms 2024-02-27T06:25:37.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:37 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:37.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:37 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:37.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:37 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:25:37.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:37 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:25:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:37 smithi150 ceph-mon[26806]: Updating smithi131:/etc/ceph/ceph.conf 2024-02-27T06:25:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:37 smithi150 ceph-mon[26806]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-27T06:25:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:37 smithi150 ceph-mon[26806]: Updating smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T06:25:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:37 smithi150 ceph-mon[26806]: Updating smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T06:25:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:37 smithi150 ceph-mon[26806]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:25:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:37 smithi150 ceph-mon[26806]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:25:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:37 smithi150 ceph-mon[26806]: Updating smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T06:25:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:37 smithi150 ceph-mon[26806]: pgmap v12: 97 pgs: 97 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 6.3 MiB/s wr, 662 op/s 2024-02-27T06:25:38.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 135982 12.10 MB/sec execute 136 sec latency 48.163 ms 2024-02-27T06:25:38.535 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:38 smithi131 ceph-mon[24834]: Updating smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T06:25:38.535 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:38 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:38.535 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:38 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:38.535 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:38 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:38.535 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:38 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:38.535 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:38 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:38.535 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:38 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:38.535 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:38 smithi131 ceph-mon[24834]: Reconfiguring prometheus.smithi131 (dependencies changed)... 2024-02-27T06:25:38.535 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:38 smithi131 ceph-mon[24834]: Reconfiguring daemon prometheus.smithi131 on smithi131 2024-02-27T06:25:38.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:38 smithi150 ceph-mon[26806]: Updating smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T06:25:38.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:38 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:38.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:38 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:38.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:38 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:38.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:38 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:38.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:38 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:38.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:38 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:38.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:38 smithi150 ceph-mon[26806]: Reconfiguring prometheus.smithi131 (dependencies changed)... 2024-02-27T06:25:38.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:38 smithi150 ceph-mon[26806]: Reconfiguring daemon prometheus.smithi131 on smithi131 2024-02-27T06:25:39.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 136457 12.06 MB/sec execute 137 sec latency 34.749 ms 2024-02-27T06:25:39.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:39 smithi131 ceph-mon[24834]: pgmap v13: 97 pgs: 97 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 438 op/s 2024-02-27T06:25:39.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:39 smithi150 ceph-mon[26806]: pgmap v13: 97 pgs: 97 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 438 op/s 2024-02-27T06:25:40.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 136843 12.05 MB/sec execute 138 sec latency 40.870 ms 2024-02-27T06:25:41.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 137322 12.06 MB/sec execute 139 sec latency 35.617 ms 2024-02-27T06:25:42.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:42 smithi131 ceph-mon[24834]: pgmap v14: 97 pgs: 97 active+clean; 329 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 667 op/s 2024-02-27T06:25:42.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 137746 12.09 MB/sec execute 140 sec latency 45.904 ms 2024-02-27T06:25:42.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:42 smithi150 ceph-mon[26806]: pgmap v14: 97 pgs: 97 active+clean; 329 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 667 op/s 2024-02-27T06:25:43.370 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:43 smithi131 ceph-mon[24834]: pgmap v15: 97 pgs: 97 active+clean; 329 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.7 MiB/s wr, 472 op/s 2024-02-27T06:25:43.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 138113 12.08 MB/sec execute 141 sec latency 36.939 ms 2024-02-27T06:25:43.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:43 smithi150 ceph-mon[26806]: pgmap v15: 97 pgs: 97 active+clean; 329 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.7 MiB/s wr, 472 op/s 2024-02-27T06:25:43.784 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:25:44.386 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:44 smithi131 ceph-mon[24834]: from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:44.511 INFO:teuthology.orchestra.run.smithi131.stdout: 5 138424 12.07 MB/sec execute 142 sec latency 72.482 ms 2024-02-27T06:25:44.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:44 smithi150 ceph-mon[26806]: from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:44.791 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:25:44.791 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (114s) 16s ago 10m 23.8M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:25:44.791 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (10m) 16s ago 10m 6405k - 17.2.0 e1d6a67b021e 5f66626dee40 2024-02-27T06:25:44.791 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (10m) 21s ago 10m 6405k - 17.2.0 e1d6a67b021e e37d494b0a36 2024-02-27T06:25:44.791 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (85s) 16s ago 10m 47.9M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:25:44.791 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (4m) 16s ago 4m 38.1M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (4m) 21s ago 4m 13.3M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (46s) 16s ago 11m 475M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283 running (2m) 21s ago 10m 70.8M - 19.0.0-1736-g635ee685 c4eb0759a73a 4e3883537bf3 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (11m) 16s ago 12m 56.4M 2048M 17.2.0 e1d6a67b021e 8e3c593dcb9f 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (10m) 21s ago 10m 45.2M 2048M 17.2.0 e1d6a67b021e 976f96599f98 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (4m) 16s ago 4m 265M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (4m) 21s ago 4m 79.6M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (82s) 16s ago 10m 10.4M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (57s) 21s ago 9m 11.0M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (9m) 16s ago 9m 462M 1936M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (8m) 16s ago 8m 409M 1936M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (8m) 16s ago 8m 288M 1936M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:25:44.792 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (7m) 16s ago 7m 422M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:25:44.793 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (7m) 21s ago 7m 439M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:25:44.793 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (7m) 21s ago 7m 494M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:25:44.793 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (6m) 21s ago 6m 419M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:25:44.793 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (6m) 21s ago 6m 573M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:25:44.793 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (59s) 16s ago 9m 55.0M - 2.33.4 514e6a882f6e 17b59b7362c7 2024-02-27T06:25:45.386 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:25:45.386 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:25:45.386 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:25:45.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:25:45.386 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:25:45.386 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:25:45.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:25:45.386 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:25:45.386 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T06:25:45.386 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:25:45.386 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:25:45.387 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:25:45.387 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:25:45.387 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:25:45.387 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-27T06:25:45.387 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:25:45.387 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:25:45.387 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:25:45.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:45 smithi131 ceph-mon[24834]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:45.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:45 smithi131 ceph-mon[24834]: from='client.24569 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:45.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:45 smithi131 ceph-mon[24834]: pgmap v16: 97 pgs: 97 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.6 MiB/s wr, 563 op/s 2024-02-27T06:25:45.508 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:45 smithi150 ceph-mon[26806]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:45.509 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:45 smithi150 ceph-mon[26806]: from='client.24569 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:45.509 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:45 smithi150 ceph-mon[26806]: pgmap v16: 97 pgs: 97 active+clean; 328 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.6 MiB/s wr, 563 op/s 2024-02-27T06:25:45.512 INFO:teuthology.orchestra.run.smithi131.stdout: 5 138694 12.03 MB/sec execute 143 sec latency 78.266 ms 2024-02-27T06:25:45.956 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:25:45.957 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:25:45.957 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:25:45.957 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:25:45.957 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:25:45.957 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr" 2024-02-27T06:25:45.957 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:25:45.957 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "2/23 daemons upgraded", 2024-02-27T06:25:45.957 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "", 2024-02-27T06:25:45.957 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:25:45.957 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:25:46.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:46 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/799723321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:25:46.421 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:46 smithi131 ceph-mon[24834]: from='client.24577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:46.519 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:46 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/799723321' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:25:46.519 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:46 smithi150 ceph-mon[26806]: from='client.24577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:25:46.520 INFO:teuthology.orchestra.run.smithi131.stdout: 5 139172 12.04 MB/sec execute 144 sec latency 38.267 ms 2024-02-27T06:25:46.630 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:25:47.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:47 smithi131 ceph-mon[24834]: from='client.? 172.21.15.131:0/3297022681' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:25:47.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:47 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:47.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:47 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:47.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:47 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:25:47.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:47 smithi131 ceph-mon[24834]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:25:47.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:47 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:25:47.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:47 smithi131 ceph-mon[24834]: pgmap v17: 97 pgs: 97 active+clean; 336 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 6.6 MiB/s wr, 671 op/s 2024-02-27T06:25:47.520 INFO:teuthology.orchestra.run.smithi131.stdout: 5 139762 12.02 MB/sec execute 145 sec latency 24.853 ms 2024-02-27T06:25:47.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:47 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/3297022681' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:25:47.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:47 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:47.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:47 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:47.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:47 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:25:47.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:47 smithi150 ceph-mon[26806]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:25:47.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:47 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:25:47.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:47 smithi150 ceph-mon[26806]: pgmap v17: 97 pgs: 97 active+clean; 336 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 6.6 MiB/s wr, 671 op/s 2024-02-27T06:25:48.520 INFO:teuthology.orchestra.run.smithi131.stdout: 5 140607 12.09 MB/sec execute 146 sec latency 23.215 ms 2024-02-27T06:25:49.185 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:48 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vvgbyr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:25:49.186 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:48 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:25:49.186 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:48 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:25:49.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:48 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vvgbyr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:25:49.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:48 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:25:49.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:48 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:25:49.520 INFO:teuthology.orchestra.run.smithi131.stdout: 5 141371 12.17 MB/sec execute 147 sec latency 25.614 ms 2024-02-27T06:25:50.193 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:49 smithi131 ceph-mon[24834]: Upgrade: Updating mgr.smithi150.vvgbyr 2024-02-27T06:25:50.193 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:49 smithi131 ceph-mon[24834]: Deploying daemon mgr.smithi150.vvgbyr on smithi150 2024-02-27T06:25:50.193 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:49 smithi131 ceph-mon[24834]: pgmap v18: 97 pgs: 97 active+clean; 336 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-02-27T06:25:50.193 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:49 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:25:50.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:49 smithi150 ceph-mon[26806]: Upgrade: Updating mgr.smithi150.vvgbyr 2024-02-27T06:25:50.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:49 smithi150 ceph-mon[26806]: Deploying daemon mgr.smithi150.vvgbyr on smithi150 2024-02-27T06:25:50.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:49 smithi150 ceph-mon[26806]: pgmap v18: 97 pgs: 97 active+clean; 336 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-02-27T06:25:50.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:49 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:25:50.520 INFO:teuthology.orchestra.run.smithi131.stdout: 5 141992 12.22 MB/sec execute 148 sec latency 45.809 ms 2024-02-27T06:25:51.520 INFO:teuthology.orchestra.run.smithi131.stdout: 5 142593 12.25 MB/sec execute 149 sec latency 48.975 ms 2024-02-27T06:25:52.227 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:52 smithi131 ceph-mon[24834]: pgmap v19: 97 pgs: 97 active+clean; 340 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 826 op/s 2024-02-27T06:25:52.428 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:52 smithi150 ceph-mon[26806]: pgmap v19: 97 pgs: 97 active+clean; 340 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 826 op/s 2024-02-27T06:25:52.520 INFO:teuthology.orchestra.run.smithi131.stdout: 5 143323 12.25 MB/sec execute 150 sec latency 70.147 ms 2024-02-27T06:25:53.244 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:53 smithi131 ceph-mon[24834]: pgmap v20: 97 pgs: 97 active+clean; 340 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 6.4 MiB/s wr, 597 op/s 2024-02-27T06:25:53.520 INFO:teuthology.orchestra.run.smithi131.stdout: 5 144050 12.31 MB/sec execute 151 sec latency 27.735 ms 2024-02-27T06:25:53.536 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:53 smithi150 ceph-mon[26806]: pgmap v20: 97 pgs: 97 active+clean; 340 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 6.4 MiB/s wr, 597 op/s 2024-02-27T06:25:54.520 INFO:teuthology.orchestra.run.smithi131.stdout: 5 144726 12.36 MB/sec execute 152 sec latency 56.435 ms 2024-02-27T06:25:55.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:55 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:55.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:55 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:55.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:55 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:25:55.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:55 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:55.264 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:55 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:55.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:55 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:55.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:55 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:55.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:55 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:25:55.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:55 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:55.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:55 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:25:55.520 INFO:teuthology.orchestra.run.smithi131.stdout: 5 145192 12.38 MB/sec execute 153 sec latency 36.042 ms 2024-02-27T06:25:56.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:56 smithi131 ceph-mon[24834]: pgmap v21: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 683 op/s 2024-02-27T06:25:56.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:56 smithi150 ceph-mon[26806]: pgmap v21: 97 pgs: 97 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 683 op/s 2024-02-27T06:25:56.520 INFO:teuthology.orchestra.run.smithi131.stdout: 5 145680 12.40 MB/sec execute 154 sec latency 40.236 ms 2024-02-27T06:25:57.520 INFO:teuthology.orchestra.run.smithi131.stdout: 5 146098 12.39 MB/sec execute 155 sec latency 44.126 ms 2024-02-27T06:25:58.166 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:57 smithi150 ceph-mon[26806]: pgmap v22: 97 pgs: 97 active+clean; 336 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 9.3 MiB/s wr, 885 op/s 2024-02-27T06:25:58.178 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:57 smithi131 ceph-mon[24834]: pgmap v22: 97 pgs: 97 active+clean; 336 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 9.3 MiB/s wr, 885 op/s 2024-02-27T06:25:58.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 146590 12.37 MB/sec execute 156 sec latency 43.410 ms 2024-02-27T06:25:59.184 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:25:59 smithi131 ceph-mon[24834]: pgmap v23: 97 pgs: 97 active+clean; 336 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.6 MiB/s wr, 750 op/s 2024-02-27T06:25:59.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 147048 12.35 MB/sec execute 157 sec latency 45.975 ms 2024-02-27T06:25:59.607 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:25:59 smithi150 ceph-mon[26806]: pgmap v23: 97 pgs: 97 active+clean; 336 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 7.6 MiB/s wr, 750 op/s 2024-02-27T06:26:00.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 147230 12.29 MB/sec execute 158 sec latency 64.018 ms 2024-02-27T06:26:00.731 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:00 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:00.731 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:00 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:00.857 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:00 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:00.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:00 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:01.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 147577 12.29 MB/sec execute 159 sec latency 63.978 ms 2024-02-27T06:26:02.270 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:02 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:02.270 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:02 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:02.270 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:02 smithi131 ceph-mon[24834]: pgmap v24: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 1.02k op/s 2024-02-27T06:26:02.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:02 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:02.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:02 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:02.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:02 smithi150 ceph-mon[26806]: pgmap v24: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 1.02k op/s 2024-02-27T06:26:02.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 147940 12.27 MB/sec execute 160 sec latency 39.782 ms 2024-02-27T06:26:03.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:03 smithi150 ceph-mon[26806]: Standby manager daemon smithi150.vvgbyr restarted 2024-02-27T06:26:03.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:03 smithi150 ceph-mon[26806]: Standby manager daemon smithi150.vvgbyr started 2024-02-27T06:26:03.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:03 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/1785131427' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/crt"}]: dispatch 2024-02-27T06:26:03.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:03 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/1785131427' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:26:03.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:03 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/1785131427' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/key"}]: dispatch 2024-02-27T06:26:03.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:03 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/1785131427' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:26:03.283 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:03 smithi131 ceph-mon[24834]: Standby manager daemon smithi150.vvgbyr restarted 2024-02-27T06:26:03.283 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:03 smithi131 ceph-mon[24834]: Standby manager daemon smithi150.vvgbyr started 2024-02-27T06:26:03.284 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:03 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/1785131427' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/crt"}]: dispatch 2024-02-27T06:26:03.284 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:03 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/1785131427' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:26:03.284 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:03 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/1785131427' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/key"}]: dispatch 2024-02-27T06:26:03.284 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:03 smithi131 ceph-mon[24834]: from='mgr.? 172.21.15.150:0/1785131427' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:26:03.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 148268 12.26 MB/sec execute 161 sec latency 48.936 ms 2024-02-27T06:26:04.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:04 smithi131 ceph-mon[24834]: mgrmap e34: smithi131.frtgqo(active, since 44s), standbys: smithi150.vvgbyr 2024-02-27T06:26:04.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:04 smithi131 ceph-mon[24834]: pgmap v25: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.5 MiB/s wr, 649 op/s 2024-02-27T06:26:04.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:04 smithi150 ceph-mon[26806]: mgrmap e34: smithi131.frtgqo(active, since 44s), standbys: smithi150.vvgbyr 2024-02-27T06:26:04.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:04 smithi150 ceph-mon[26806]: pgmap v25: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.5 MiB/s wr, 649 op/s 2024-02-27T06:26:04.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 148828 12.29 MB/sec execute 162 sec latency 26.245 ms 2024-02-27T06:26:05.271 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:05 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:26:05.271 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:05 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:05.271 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:05 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:05.306 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:05 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:26:05.306 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:05 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:05.306 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:05 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:05.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 149383 12.32 MB/sec execute 163 sec latency 34.451 ms 2024-02-27T06:26:06.312 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: pgmap v26: 97 pgs: 97 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 674 op/s 2024-02-27T06:26:06.312 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:06.312 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:06.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:26:06.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:26:06.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:06.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:06.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:26:06.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:26:06.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:06.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi131.frtgqo"}]: dispatch 2024-02-27T06:26:06.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi131.frtgqo"}]': finished 2024-02-27T06:26:06.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi150.vvgbyr"}]: dispatch 2024-02-27T06:26:06.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi150.vvgbyr"}]': finished 2024-02-27T06:26:06.313 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:06 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: pgmap v26: 97 pgs: 97 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 674 op/s 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi131.frtgqo"}]: dispatch 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi131.frtgqo"}]': finished 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi150.vvgbyr"}]: dispatch 2024-02-27T06:26:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi150.vvgbyr"}]': finished 2024-02-27T06:26:06.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:06 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T06:26:06.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 149776 12.32 MB/sec execute 164 sec latency 48.333 ms 2024-02-27T06:26:07.086 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:07 smithi131 ceph-mon[24834]: Upgrade: Setting container_image for all mgr 2024-02-27T06:26:07.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:07 smithi150 ceph-mon[26806]: Upgrade: Setting container_image for all mgr 2024-02-27T06:26:07.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 150136 12.28 MB/sec execute 165 sec latency 53.262 ms 2024-02-27T06:26:08.091 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:08 smithi131 ceph-mon[24834]: pgmap v27: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 7.9 MiB/s wr, 795 op/s 2024-02-27T06:26:08.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:08 smithi150 ceph-mon[26806]: pgmap v27: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 7.9 MiB/s wr, 795 op/s 2024-02-27T06:26:08.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 150537 12.24 MB/sec execute 166 sec latency 44.153 ms 2024-02-27T06:26:09.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 150810 12.22 MB/sec execute 167 sec latency 58.548 ms 2024-02-27T06:26:10.118 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:10 smithi131 ceph-mon[24834]: pgmap v28: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.2 MiB/s wr, 503 op/s 2024-02-27T06:26:10.119 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:10 smithi131 ceph-mon[24834]: Upgrade: Updating mon.smithi131 2024-02-27T06:26:10.119 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:10 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:10.119 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:10 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:26:10.119 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:10 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T06:26:10.119 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:10 smithi131 ceph-mon[24834]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:26:10.119 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:10 smithi131 ceph-mon[24834]: Deploying daemon mon.smithi131 on smithi131 2024-02-27T06:26:10.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:10 smithi150 ceph-mon[26806]: pgmap v28: 97 pgs: 97 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.2 MiB/s wr, 503 op/s 2024-02-27T06:26:10.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:10 smithi150 ceph-mon[26806]: Upgrade: Updating mon.smithi131 2024-02-27T06:26:10.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:10 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:10.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:10 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:26:10.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:10 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T06:26:10.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:10 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:26:10.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:10 smithi150 ceph-mon[26806]: Deploying daemon mon.smithi131 on smithi131 2024-02-27T06:26:10.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 151288 12.22 MB/sec execute 168 sec latency 59.144 ms 2024-02-27T06:26:11.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 151690 12.23 MB/sec execute 169 sec latency 53.930 ms 2024-02-27T06:26:12.141 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:12 smithi131 ceph-mon[24834]: pgmap v29: 97 pgs: 97 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 752 op/s 2024-02-27T06:26:12.357 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:12 smithi150 ceph-mon[26806]: pgmap v29: 97 pgs: 97 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 752 op/s 2024-02-27T06:26:12.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 152039 12.22 MB/sec execute 170 sec latency 50.136 ms 2024-02-27T06:26:12.648 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:12 smithi131 systemd[1]: Stopping Ceph mon.smithi131 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:26:13.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:12 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131[24830]: 2024-02-27T06:26:12.951+0000 7f4bd9d4c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi131 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:26:13.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:12 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131[24830]: 2024-02-27T06:26:12.951+0000 7f4bd9d4c700 -1 mon.smithi131@0(leader) e2 *** Got Signal Terminated *** 2024-02-27T06:26:13.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:12 smithi131 podman[66091]: 2024-02-27 06:26:12.983764738 +0000 UTC m=+0.237035033 container died 8e3c593dcb9f7fcf100b2019e24ea771949230f151d3723b649cf3ac9eed94f4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, RELEASE=HEAD, release=754, vcs-type=git, GIT_BRANCH=HEAD, io.openshift.expose-services=, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8) 2024-02-27T06:26:13.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 152374 12.21 MB/sec execute 171 sec latency 51.272 ms 2024-02-27T06:26:13.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:13 smithi131 podman[66091]: 2024-02-27 06:26:13.577664468 +0000 UTC m=+0.830934764 container cleanup 8e3c593dcb9f7fcf100b2019e24ea771949230f151d3723b649cf3ac9eed94f4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, name=centos-stream, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , ceph=True, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, distribution-scope=public, RELEASE=HEAD, vcs-type=git, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.buildah.version=1.19.8, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-27T06:26:13.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:13 smithi131 bash[66091]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131 2024-02-27T06:26:14.176 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:13 smithi131 podman[66114]: 2024-02-27 06:26:13.993809645 +0000 UTC m=+1.003217407 container remove 8e3c593dcb9f7fcf100b2019e24ea771949230f151d3723b649cf3ac9eed94f4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, GIT_CLEAN=True, io.openshift.expose-services=, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, distribution-scope=public, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux ) 2024-02-27T06:26:14.428 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:14 smithi131 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi131.service: Deactivated successfully. 2024-02-27T06:26:14.428 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:14 smithi131 systemd[1]: Stopped Ceph mon.smithi131 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:26:14.428 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:14 smithi131 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi131.service: Consumed 16.397s CPU time. 2024-02-27T06:26:14.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 152758 12.22 MB/sec execute 172 sec latency 44.704 ms 2024-02-27T06:26:14.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:14 smithi131 systemd[1]: Starting Ceph mon.smithi131 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:26:14.931 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:14 smithi131 podman[66239]: 2024-02-27 06:26:14.729116342 +0000 UTC m=+0.027420578 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:26:14.931 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:14 smithi131 podman[66239]: 2024-02-27 06:26:14.840889025 +0000 UTC m=+0.139193252 container create e5901645d10a3889439178583676be37341400fe297523827b332aa80c45efde (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:26:15.437 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 podman[66239]: 2024-02-27 06:26:15.378052541 +0000 UTC m=+0.676356772 container init e5901645d10a3889439178583676be37341400fe297523827b332aa80c45efde (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-02-27T06:26:15.437 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 podman[66239]: 2024-02-27 06:26:15.38317906 +0000 UTC m=+0.681483289 container start e5901645d10a3889439178583676be37341400fe297523827b332aa80c45efde (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:26:15.437 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: set uid:gid to 167:167 (ceph:ceph) 2024-02-27T06:26:15.437 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev), process ceph-mon, pid 2 2024-02-27T06:26:15.437 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: pidfile_write: ignore empty --pid-file 2024-02-27T06:26:15.437 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: load: jerasure load: lrc 2024-02-27T06:26:15.437 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: RocksDB version: 7.9.2 2024-02-27T06:26:15.437 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Git sha 0 2024-02-27T06:26:15.437 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Compile date 2024-02-26 21:43:15 2024-02-27T06:26:15.437 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: DB SUMMARY 2024-02-27T06:26:15.437 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: DB Session ID: C2OU3CBR0YLHZWF36J1S 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: CURRENT file: CURRENT 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: IDENTITY file: IDENTITY 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: MANIFEST file: MANIFEST-000015 size: 978 Bytes 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi131/store.db dir, Total Num: 1, files: 000030.sst 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi131/store.db: 000028.log size: 2261344 ; 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.error_if_exists: 0 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.create_if_missing: 0 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.paranoid_checks: 1 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.env: 0x5600945b1b20 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.fs: PosixFileSystem 2024-02-27T06:26:15.438 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.info_log: 0x560096d3f780 2024-02-27T06:26:15.439 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_file_opening_threads: 16 2024-02-27T06:26:15.439 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.statistics: (nil) 2024-02-27T06:26:15.439 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.use_fsync: 0 2024-02-27T06:26:15.439 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_log_file_size: 0 2024-02-27T06:26:15.439 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.keep_log_file_num: 1000 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.recycle_log_file_num: 0 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.allow_fallocate: 1 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.allow_mmap_reads: 0 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.allow_mmap_writes: 0 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.use_direct_reads: 0 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.create_missing_column_families: 0 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.db_log_dir: 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.wal_dir: 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-27T06:26:15.440 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.advise_random_on_open: 1 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.db_write_buffer_size: 0 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.write_buffer_manager: 0x560096d13400 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.rate_limiter: (nil) 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.wal_recovery_mode: 2 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.enable_thread_tracking: 0 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.enable_pipelined_write: 0 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.unordered_write: 0 2024-02-27T06:26:15.441 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.row_cache: None 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.wal_filter: None 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.allow_ingest_behind: 0 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.two_write_queues: 0 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.manual_wal_flush: 0 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.wal_compression: 0 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.atomic_flush: 0 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-27T06:26:15.442 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.log_readahead_size: 0 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.best_efforts_recovery: 0 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.allow_data_in_errors: 0 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.db_host_id: __hostname__ 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_background_jobs: 2 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_background_compactions: -1 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_subcompactions: 1 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_total_wal_size: 0 2024-02-27T06:26:15.443 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_open_files: -1 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bytes_per_sync: 0 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_readahead_size: 0 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_background_flushes: -1 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Compression algorithms supported: 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: kZSTD supported: 0 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: kXpressCompression supported: 0 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: kLZ4HCCompression supported: 1 2024-02-27T06:26:15.444 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: kZlibCompression supported: 1 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: kSnappyCompression supported: 1 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: kLZ4Compression supported: 1 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: kBZip2Compression supported: 0 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi131/store.db/MANIFEST-000015 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.merge_operator: 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_filter: None 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_filter_factory: None 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.sst_partitioner_factory: None 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-27T06:26:15.445 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560096d3f8a0) 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: cache_index_and_filter_blocks: 1 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: pin_top_level_index_and_filter: 1 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: index_type: 0 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: data_block_index_type: 0 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: index_shortening: 1 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: checksum: 4 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: no_block_cache: 0 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_cache: 0x560096d43090 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_cache_name: BinnedLRUCache 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_cache_options: 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: capacity : 536870912 2024-02-27T06:26:15.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: num_shard_bits : 4 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: strict_capacity_limit : 0 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: high_pri_pool_ratio: 0.000 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_cache_compressed: (nil) 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: persistent_cache: (nil) 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_size: 4096 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_size_deviation: 10 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_restart_interval: 16 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: index_block_restart_interval: 1 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: metadata_block_size: 4096 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: partition_filters: 0 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: use_delta_encoding: 1 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: filter_policy: bloomfilter 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: whole_key_filtering: 1 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: verify_compression: 0 2024-02-27T06:26:15.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: read_amp_bytes_per_bit: 0 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: format_version: 5 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: enable_index_compression: 1 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: block_align: 0 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: max_auto_readahead_size: 262144 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: prepopulate_block_cache: 0 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: initial_auto_readahead_size: 8192 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: num_file_reads_for_auto_readahead: 2 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.write_buffer_size: 33554432 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_write_buffer_number: 2 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compression: NoCompression 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bottommost_compression: Disabled 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.prefix_extractor: nullptr 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.num_levels: 7 2024-02-27T06:26:15.448 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compression_opts.level: 32767 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compression_opts.strategy: 0 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-27T06:26:15.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compression_opts.enabled: false 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.target_file_size_base: 67108864 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-27T06:26:15.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-27T06:26:15.451 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-27T06:26:15.451 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-27T06:26:15.451 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-27T06:26:15.451 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-27T06:26:15.451 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.arena_block_size: 1048576 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.disable_auto_compactions: 0 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-27T06:26:15.452 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.table_properties_collectors: 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.inplace_update_support: 0 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.bloom_locality: 0 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.max_successive_merges: 0 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.paranoid_file_checks: 0 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.force_consistency_checks: 1 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.report_bg_io_stats: 0 2024-02-27T06:26:15.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.ttl: 2592000 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.enable_blob_files: false 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.min_blob_size: 0 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.blob_file_size: 268435456 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.blob_file_starting_level: 0 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-27T06:26:15.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi131/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11847, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f83a36e6-548c-4317-bc8a-6137ab43e9d0 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709015175418321, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709015175426605, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2077688, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11841, "largest_seqno": 12882, "table_properties": {"data_size": 2072416, "index_size": 3105, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1157, "raw_key_size": 10299, "raw_average_key_size": 23, "raw_value_size": 2062610, "raw_average_value_size": 4624, "num_data_blocks": 143, "num_entries": 446, "num_filter_entries": 446, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709015175, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f83a36e6-548c-4317-bc8a-6137ab43e9d0", "db_session_id": "C2OU3CBR0YLHZWF36J1S", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709015175426717, "job": 1, "event": "recovery_finished"} 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi131/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560096e32000 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: DB pointer 0x560096e1e000 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ** DB Stats ** 2024-02-27T06:26:15.455 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ** Compaction Stats [default] ** 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: L0 1/0 1.98 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 517.1 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: L6 1/0 6.64 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Sum 2/0 8.62 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 517.1 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-27T06:26:15.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 517.1 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ** Compaction Stats [default] ** 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 517.1 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T06:26:15.457 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Cumulative compaction: 0.00 GB write, 151.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Interval compaction: 0.00 GB write, 151.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Block cache BinnedLRUCache@0x560096d43090#2 capacity: 512.00 MB usage: 33.80 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Block cache entry stats(count,size,portion): DataBlock(1,4.08 KB,0.000777841%) FilterBlock(2,10.31 KB,0.00196695%) IndexBlock(2,19.41 KB,0.00370145%) Misc(1,0.00 KB,0%) 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: starting mon.smithi131 rank 0 at public addrs [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] at bind addrs [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi131 fsid 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: mon.smithi131@-1(???) e2 preinit fsid 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: mon.smithi131@-1(???).mds e7 new map 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: mon.smithi131@-1(???).mds e7 print_map 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: e7 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: legacy client fscid: 1 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:26:15.458 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Filesystem 'foofs' (1) 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: fs_name foofs 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: epoch 5 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: created 2024-02-27T06:20:44.948420+0000 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: modified 2024-02-27T06:20:50.974048+0000 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: tableserver 0 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: root 0 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: session_timeout 60 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: session_autoclose 300 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: max_file_size 1099511627776 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: max_xattr_size 65536 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: required_client_features {} 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: last_failure 0 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: last_failure_osd_epoch 0 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-27T06:26:15.459 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: max_mds 1 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: in 0 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: up {0=14476} 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: failed 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: damaged 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: stopped 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: data_pools [3] 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: metadata_pool 2 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: inline_data disabled 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: balancer 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: bal_rank_mask -1 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: standby_count_wanted 1 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: [mds.foofs.smithi131.mhuwwb{0:14476} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.131:6834/3029043434,v1:172.21.15.131:6835/3029043434] compat {c=[1],r=[1],i=[7ff]}] 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:26:15.460 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:26:15.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: Standby daemons: 2024-02-27T06:26:15.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: 2024-02-27T06:26:15.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout: [mds.foofs.smithi150.vsnjhu{-1:24273} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.150:6832/3371494173,v1:172.21.15.150:6833/3371494173] compat {c=[1],r=[1],i=[7ff]}] 2024-02-27T06:26:15.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: mon.smithi131@-1(???).osd e72 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-27T06:26:15.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: mon.smithi131@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T06:26:15.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: mon.smithi131@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T06:26:15.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: mon.smithi131@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T06:26:15.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 ceph-mon[66259]: mon.smithi131@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-02-27T06:26:15.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 153119 12.19 MB/sec execute 173 sec latency 32.678 ms 2024-02-27T06:26:15.687 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 bash[66239]: e5901645d10a3889439178583676be37341400fe297523827b332aa80c45efde 2024-02-27T06:26:15.687 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:15 smithi131 systemd[1]: Started Ceph mon.smithi131 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:26:16.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:26:16.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:26:16.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: pgmap v30: 97 pgs: 97 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.8 MiB/s wr, 481 op/s 2024-02-27T06:26:16.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: pgmap v31: 97 pgs: 97 active+clean; 328 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.3 MiB/s wr, 504 op/s 2024-02-27T06:26:16.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: mon.smithi131 calling monitor election 2024-02-27T06:26:16.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: mon.smithi131 is new leader, mons smithi131,smithi150 in quorum (ranks 0,1) 2024-02-27T06:26:16.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: monmap e2: 2 mons at {smithi131=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],smithi150=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T06:26:16.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 1 up:standby 2024-02-27T06:26:16.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: osdmap e72: 8 total, 8 up, 8 in 2024-02-27T06:26:16.190 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: mgrmap e34: smithi131.frtgqo(active, since 57s), standbys: smithi150.vvgbyr 2024-02-27T06:26:16.191 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: overall HEALTH_OK 2024-02-27T06:26:16.191 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: from='mgr.24515 ' entity='' 2024-02-27T06:26:16.191 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:16 smithi131 ceph-mon[66259]: mgrmap e35: smithi131.frtgqo(active, since 57s), standbys: smithi150.vvgbyr 2024-02-27T06:26:16.521 INFO:teuthology.orchestra.run.smithi131.stdout: 5 153641 12.19 MB/sec execute 174 sec latency 45.629 ms 2024-02-27T06:26:16.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:26:16.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: from='mgr.24515 172.21.15.131:0/2939702222' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:26:16.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: pgmap v30: 97 pgs: 97 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.8 MiB/s wr, 481 op/s 2024-02-27T06:26:16.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: pgmap v31: 97 pgs: 97 active+clean; 328 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.3 MiB/s wr, 504 op/s 2024-02-27T06:26:16.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: mon.smithi131 calling monitor election 2024-02-27T06:26:16.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: mon.smithi131 is new leader, mons smithi131,smithi150 in quorum (ranks 0,1) 2024-02-27T06:26:16.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: monmap e2: 2 mons at {smithi131=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],smithi150=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T06:26:16.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 1 up:standby 2024-02-27T06:26:16.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: osdmap e72: 8 total, 8 up, 8 in 2024-02-27T06:26:16.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: mgrmap e34: smithi131.frtgqo(active, since 57s), standbys: smithi150.vvgbyr 2024-02-27T06:26:16.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: overall HEALTH_OK 2024-02-27T06:26:16.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: from='mgr.24515 ' entity='' 2024-02-27T06:26:16.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:16 smithi150 ceph-mon[26806]: mgrmap e35: smithi131.frtgqo(active, since 57s), standbys: smithi150.vvgbyr 2024-02-27T06:26:17.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 154037 12.15 MB/sec execute 175 sec latency 41.523 ms 2024-02-27T06:26:18.216 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:18 smithi131 ceph-mon[66259]: mgrmap e36: smithi131.frtgqo(active, since 58s), standbys: smithi150.vvgbyr 2024-02-27T06:26:18.502 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:18 smithi150 ceph-mon[26806]: mgrmap e36: smithi131.frtgqo(active, since 58s), standbys: smithi150.vvgbyr 2024-02-27T06:26:18.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 154419 12.14 MB/sec execute 176 sec latency 53.586 ms 2024-02-27T06:26:19.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 154901 12.15 MB/sec execute 177 sec latency 37.477 ms 2024-02-27T06:26:20.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 155275 12.15 MB/sec execute 178 sec latency 56.141 ms 2024-02-27T06:26:21.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 155693 12.16 MB/sec execute 179 sec latency 37.410 ms 2024-02-27T06:26:22.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 156005 12.15 MB/sec execute 180 sec latency 39.237 ms 2024-02-27T06:26:23.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 156353 12.14 MB/sec execute 181 sec latency 35.115 ms 2024-02-27T06:26:24.067 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:23 smithi131 ceph-mon[66259]: from='mgr.? 172.21.15.150:0/2766736834' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/crt"}]: dispatch 2024-02-27T06:26:24.067 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:23 smithi131 ceph-mon[66259]: from='mgr.? 172.21.15.150:0/2766736834' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:26:24.067 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:23 smithi131 ceph-mon[66259]: from='mgr.? 172.21.15.150:0/2766736834' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/key"}]: dispatch 2024-02-27T06:26:24.067 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:23 smithi131 ceph-mon[66259]: from='mgr.? 172.21.15.150:0/2766736834' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:26:24.067 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:23 smithi131 ceph-mon[66259]: Standby manager daemon smithi150.vvgbyr restarted 2024-02-27T06:26:24.068 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:23 smithi131 ceph-mon[66259]: Standby manager daemon smithi150.vvgbyr started 2024-02-27T06:26:24.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:23 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/2766736834' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/crt"}]: dispatch 2024-02-27T06:26:24.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:23 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/2766736834' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-27T06:26:24.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:23 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/2766736834' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi150.vvgbyr/key"}]: dispatch 2024-02-27T06:26:24.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:23 smithi150 ceph-mon[26806]: from='mgr.? 172.21.15.150:0/2766736834' entity='mgr.smithi150.vvgbyr' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-27T06:26:24.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:23 smithi150 ceph-mon[26806]: Standby manager daemon smithi150.vvgbyr restarted 2024-02-27T06:26:24.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:23 smithi150 ceph-mon[26806]: Standby manager daemon smithi150.vvgbyr started 2024-02-27T06:26:24.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 156600 12.11 MB/sec execute 182 sec latency 47.179 ms 2024-02-27T06:26:25.022 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:24 smithi131 ceph-mon[66259]: mgrmap e37: smithi131.frtgqo(active, since 65s), standbys: smithi150.vvgbyr 2024-02-27T06:26:25.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:24 smithi150 ceph-mon[26806]: mgrmap e37: smithi131.frtgqo(active, since 65s), standbys: smithi150.vvgbyr 2024-02-27T06:26:25.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 156940 12.09 MB/sec execute 183 sec latency 58.692 ms 2024-02-27T06:26:26.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 157256 12.05 MB/sec execute 184 sec latency 51.467 ms 2024-02-27T06:26:26.548 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: Active manager daemon smithi131.frtgqo restarted 2024-02-27T06:26:26.548 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: Activating manager daemon smithi131.frtgqo 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: osdmap e73: 8 total, 8 up, 8 in 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: mgrmap e38: smithi131.frtgqo(active, starting, since 0.0234252s), standbys: smithi150.vvgbyr 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi131.frtgqo", "id": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi150.vvgbyr", "id": "smithi150.vvgbyr"}]: dispatch 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:26:26.549 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:26:26.550 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:26:26.550 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:26:26.550 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T06:26:26.550 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T06:26:26.550 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T06:26:26.648 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: Active manager daemon smithi131.frtgqo restarted 2024-02-27T06:26:26.648 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: Activating manager daemon smithi131.frtgqo 2024-02-27T06:26:26.648 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: osdmap e73: 8 total, 8 up, 8 in 2024-02-27T06:26:26.648 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: mgrmap e38: smithi131.frtgqo(active, starting, since 0.0234252s), standbys: smithi150.vvgbyr 2024-02-27T06:26:26.648 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:26:26.648 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:26:26.648 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:26:26.648 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi131.frtgqo", "id": "smithi131.frtgqo"}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr metadata", "who": "smithi150.vvgbyr", "id": "smithi150.vvgbyr"}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-27T06:26:26.649 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:26 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-27T06:26:27.472 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:26:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 157549 12.01 MB/sec execute 185 sec latency 53.725 ms 2024-02-27T06:26:27.579 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:27 smithi131 ceph-mon[66259]: Manager daemon smithi131.frtgqo is now available 2024-02-27T06:26:27.579 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:26:27.579 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:26:27.579 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:26:27.579 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:26:27.579 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:27 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:26:27.579 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:26:27.579 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:27 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:26:27.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:27 smithi150 ceph-mon[26806]: Manager daemon smithi131.frtgqo is now available 2024-02-27T06:26:27.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:27 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:26:27.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:27 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:26:27.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:27 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:26:27.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:27 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:26:27.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:27 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/mirror_snapshot_schedule"}]: dispatch 2024-02-27T06:26:27.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:27 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:26:27.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:27 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi131.frtgqo/trash_purge_schedule"}]: dispatch 2024-02-27T06:26:28.521 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (2m) 24s ago 11m 23.8M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (11m) 24s ago 11m 6405k - 17.2.0 e1d6a67b021e 5f66626dee40 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (10m) 28s ago 10m 6405k - 17.2.0 e1d6a67b021e e37d494b0a36 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (2m) 24s ago 11m 47.5M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (5m) 24s ago 5m 38.6M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (5m) 28s ago 5m 13.5M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (90s) 24s ago 12m 493M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (35s) 28s ago 10m 92.1M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (12m) 24s ago 12m 56.5M 2048M 17.2.0 e1d6a67b021e 8e3c593dcb9f 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (10m) 28s ago 10m 42.6M 2048M 17.2.0 e1d6a67b021e 976f96599f98 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (5m) 24s ago 5m 269M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (5m) 28s ago 5m 81.1M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (2m) 24s ago 11m 13.2M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:26:28.522 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (100s) 28s ago 10m 19.0M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:26:28.523 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (10m) 24s ago 10m 500M 1936M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:26:28.523 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (9m) 24s ago 9m 439M 1936M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:26:28.523 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (9m) 24s ago 9m 327M 1936M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:26:28.523 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (8m) 24s ago 8m 460M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:26:28.523 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (8m) 28s ago 8m 480M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:26:28.523 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (7m) 28s ago 7m 562M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:26:28.523 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (7m) 28s ago 7m 474M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:26:28.523 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (7m) 28s ago 7m 365M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:26:28.523 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (42s) 24s ago 10m 49.2M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:26:28.523 INFO:teuthology.orchestra.run.smithi131.stdout: 5 157759 11.97 MB/sec execute 186 sec latency 60.559 ms 2024-02-27T06:26:28.619 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:28 smithi131 ceph-mon[66259]: mgrmap e39: smithi131.frtgqo(active, since 1.07477s), standbys: smithi150.vvgbyr 2024-02-27T06:26:28.619 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:28 smithi131 ceph-mon[66259]: from='client.24593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:26:28.619 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:28 smithi131 ceph-mon[66259]: pgmap v3: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:26:28.619 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:28 smithi131 ceph-mon[66259]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:26:28.781 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:28 smithi150 ceph-mon[26806]: mgrmap e39: smithi131.frtgqo(active, since 1.07477s), standbys: smithi150.vvgbyr 2024-02-27T06:26:28.782 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:28 smithi150 ceph-mon[26806]: from='client.24593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:26:28.782 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:28 smithi150 ceph-mon[26806]: pgmap v3: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:26:28.782 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:28 smithi150 ceph-mon[26806]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:26:29.148 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:26:29.148 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:26:29.148 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-27T06:26:29.148 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T06:26:29.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:26:29.148 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:26:29.148 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:26:29.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:26:29.148 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:26:29.148 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T06:26:29.148 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:26:29.148 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:26:29.149 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:26:29.149 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:26:29.149 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:26:29.149 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-02-27T06:26:29.149 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T06:26:29.149 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:26:29.149 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:26:29.522 INFO:teuthology.orchestra.run.smithi131.stdout: 5 158049 11.96 MB/sec execute 187 sec latency 76.633 ms 2024-02-27T06:26:29.634 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:29 smithi131 ceph-mon[66259]: pgmap v4: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:26:29.634 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:29 smithi131 ceph-mon[66259]: mgrmap e40: smithi131.frtgqo(active, since 2s), standbys: smithi150.vvgbyr 2024-02-27T06:26:29.634 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:29 smithi131 ceph-mon[66259]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:26:29.635 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:29 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/431802316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:26:29.690 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:26:29.690 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:26:29.690 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:26:29.690 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:26:29.690 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:26:29.690 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr" 2024-02-27T06:26:29.691 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:26:29.691 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "2/23 daemons upgraded", 2024-02-27T06:26:29.691 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "", 2024-02-27T06:26:29.691 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:26:29.691 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:26:29.720 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:29 smithi150 ceph-mon[26806]: pgmap v4: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:26:29.720 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:29 smithi150 ceph-mon[26806]: mgrmap e40: smithi131.frtgqo(active, since 2s), standbys: smithi150.vvgbyr 2024-02-27T06:26:29.720 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:29 smithi150 ceph-mon[26806]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:26:29.720 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:29 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/431802316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:26:30.305 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:26:30.523 INFO:teuthology.orchestra.run.smithi131.stdout: 5 158606 11.98 MB/sec execute 188 sec latency 33.522 ms 2024-02-27T06:26:30.639 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:30 smithi131 ceph-mon[66259]: [27/Feb/2024:06:26:29] ENGINE Bus STARTING 2024-02-27T06:26:30.640 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:30 smithi131 ceph-mon[66259]: [27/Feb/2024:06:26:29] ENGINE Serving on http://172.21.15.131:8765 2024-02-27T06:26:30.640 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:30 smithi131 ceph-mon[66259]: [27/Feb/2024:06:26:29] ENGINE Serving on https://172.21.15.131:7150 2024-02-27T06:26:30.640 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:30 smithi131 ceph-mon[66259]: [27/Feb/2024:06:26:29] ENGINE Bus STARTED 2024-02-27T06:26:30.640 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:30 smithi131 ceph-mon[66259]: from='client.34144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:26:30.640 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:30 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/484533892' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:26:30.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:30 smithi150 ceph-mon[26806]: [27/Feb/2024:06:26:29] ENGINE Bus STARTING 2024-02-27T06:26:30.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:30 smithi150 ceph-mon[26806]: [27/Feb/2024:06:26:29] ENGINE Serving on http://172.21.15.131:8765 2024-02-27T06:26:30.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:30 smithi150 ceph-mon[26806]: [27/Feb/2024:06:26:29] ENGINE Serving on https://172.21.15.131:7150 2024-02-27T06:26:30.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:30 smithi150 ceph-mon[26806]: [27/Feb/2024:06:26:29] ENGINE Bus STARTED 2024-02-27T06:26:30.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:30 smithi150 ceph-mon[26806]: from='client.34144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:26:30.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:30 smithi150 ceph-mon[26806]: from='client.? 172.21.15.131:0/484533892' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:26:31.523 INFO:teuthology.orchestra.run.smithi131.stdout: 5 159097 12.00 MB/sec execute 189 sec latency 23.901 ms 2024-02-27T06:26:31.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:31 smithi131 ceph-mon[66259]: pgmap v5: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:26:31.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:31 smithi131 ceph-mon[66259]: mgrmap e41: smithi131.frtgqo(active, since 4s), standbys: smithi150.vvgbyr 2024-02-27T06:26:31.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:31 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:31.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:31 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:31.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:31 smithi150 ceph-mon[26806]: pgmap v5: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:26:31.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:31 smithi150 ceph-mon[26806]: mgrmap e41: smithi131.frtgqo(active, since 4s), standbys: smithi150.vvgbyr 2024-02-27T06:26:31.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:31 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:31.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:31 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:32.523 INFO:teuthology.orchestra.run.smithi131.stdout: 5 159510 12.00 MB/sec execute 190 sec latency 39.118 ms 2024-02-27T06:26:33.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:33 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:33.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:33 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:33.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:33 smithi131 ceph-mon[66259]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:26:33.523 INFO:teuthology.orchestra.run.smithi131.stdout: 5 160052 12.02 MB/sec execute 191 sec latency 25.565 ms 2024-02-27T06:26:33.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:33 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:33.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:33 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:33.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:33 smithi150 ceph-mon[26806]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-02-27T06:26:34.523 INFO:teuthology.orchestra.run.smithi131.stdout: 5 160481 12.03 MB/sec execute 192 sec latency 52.463 ms 2024-02-27T06:26:35.523 INFO:teuthology.orchestra.run.smithi131.stdout: 5 160893 11.99 MB/sec execute 193 sec latency 49.232 ms 2024-02-27T06:26:35.692 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:35 smithi131 ceph-mon[66259]: pgmap v7: 97 pgs: 97 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 34 KiB/s rd, 4.3 MiB/s wr, 419 op/s 2024-02-27T06:26:35.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:35 smithi150 ceph-mon[26806]: pgmap v7: 97 pgs: 97 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 34 KiB/s rd, 4.3 MiB/s wr, 419 op/s 2024-02-27T06:26:36.523 INFO:teuthology.orchestra.run.smithi131.stdout: 5 161280 11.97 MB/sec execute 194 sec latency 39.897 ms 2024-02-27T06:26:37.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:37 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:37.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:37 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:37.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:37 smithi131 ceph-mon[66259]: pgmap v8: 97 pgs: 97 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 3.4 MiB/s wr, 325 op/s 2024-02-27T06:26:37.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:37 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:37.461 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:37 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:37.523 INFO:teuthology.orchestra.run.smithi131.stdout: 5 161689 11.97 MB/sec execute 195 sec latency 49.147 ms 2024-02-27T06:26:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:37 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:37 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:37 smithi150 ceph-mon[26806]: pgmap v8: 97 pgs: 97 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 3.4 MiB/s wr, 325 op/s 2024-02-27T06:26:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:37 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:37.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:37 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:38.523 INFO:teuthology.orchestra.run.smithi131.stdout: 5 162062 11.95 MB/sec execute 196 sec latency 43.989 ms 2024-02-27T06:26:39.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:39 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:39.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:39 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:39.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:39 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:26:39.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:39 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:26:39.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:39 smithi131 ceph-mon[66259]: Updating smithi131:/etc/ceph/ceph.conf 2024-02-27T06:26:39.254 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:39 smithi131 ceph-mon[66259]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-27T06:26:39.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 162379 11.96 MB/sec execute 197 sec latency 83.789 ms 2024-02-27T06:26:39.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:39 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:39.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:39 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:39.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:39 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:26:39.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:39 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:26:39.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:39 smithi150 ceph-mon[26806]: Updating smithi131:/etc/ceph/ceph.conf 2024-02-27T06:26:39.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:39 smithi150 ceph-mon[26806]: Updating smithi150:/etc/ceph/ceph.conf 2024-02-27T06:26:40.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: pgmap v9: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.4 MiB/s wr, 559 op/s 2024-02-27T06:26:40.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: Updating smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T06:26:40.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: Updating smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T06:26:40.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:26:40.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:26:40.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: Updating smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T06:26:40.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: Updating smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T06:26:40.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:40.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:40.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:40.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:40.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:40.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:40.282 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:26:40.282 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:26:40.282 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:40 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T06:26:40.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 162703 11.94 MB/sec execute 198 sec latency 47.308 ms 2024-02-27T06:26:40.564 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: pgmap v9: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 5.4 MiB/s wr, 559 op/s 2024-02-27T06:26:40.564 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: Updating smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: Updating smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.conf 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: Updating smithi150:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: Updating smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: Updating smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/config/ceph.client.admin.keyring 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:26:40.565 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:40 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-27T06:26:41.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 163035 11.93 MB/sec execute 199 sec latency 38.129 ms 2024-02-27T06:26:42.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:42 smithi131 ceph-mon[66259]: pgmap v10: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.9 MiB/s wr, 508 op/s 2024-02-27T06:26:42.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:26:42.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:42 smithi131 ceph-mon[66259]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:42.304 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:26:42.305 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T06:26:42.305 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:26:42.462 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:42 smithi150 ceph-mon[26806]: pgmap v10: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.9 MiB/s wr, 508 op/s 2024-02-27T06:26:42.462 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:42 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:26:42.463 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:42 smithi150 ceph-mon[26806]: from='mgr.34102 ' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:42.463 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:42 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:26:42.463 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:42 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T06:26:42.463 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:42 smithi150 ceph-mon[26806]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:26:42.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 163382 11.93 MB/sec execute 200 sec latency 50.301 ms 2024-02-27T06:26:43.331 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:43 smithi131 ceph-mon[66259]: Upgrade: Updating mon.smithi150 2024-02-27T06:26:43.331 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:43 smithi131 ceph-mon[66259]: Deploying daemon mon.smithi150 on smithi150 2024-02-27T06:26:43.462 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:43 smithi150 ceph-mon[26806]: Upgrade: Updating mon.smithi150 2024-02-27T06:26:43.463 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:43 smithi150 ceph-mon[26806]: Deploying daemon mon.smithi150 on smithi150 2024-02-27T06:26:43.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 163628 11.90 MB/sec execute 201 sec latency 51.684 ms 2024-02-27T06:26:43.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:43 smithi150 systemd[1]: Stopping Ceph mon.smithi150 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:26:44.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:44 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150[26802]: 2024-02-27T06:26:44.002+0000 7fb683cb5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi150 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:26:44.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:44 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150[26802]: 2024-02-27T06:26:44.002+0000 7fb683cb5700 -1 mon.smithi150@1(peon) e2 *** Got Signal Terminated *** 2024-02-27T06:26:44.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:44 smithi150 podman[52646]: 2024-02-27 06:26:44.176526443 +0000 UTC m=+0.304660860 container died 976f96599f98d82552bd913ce1b6135c7ce3b0a452a56e634e3dad6fa75559f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150, RELEASE=HEAD, com.redhat.component=centos-stream-container, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, ceph=True, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, release=754, version=8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream) 2024-02-27T06:26:44.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 163959 11.89 MB/sec execute 202 sec latency 60.237 ms 2024-02-27T06:26:44.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:44 smithi150 podman[52646]: 2024-02-27 06:26:44.570234237 +0000 UTC m=+0.698368654 container cleanup 976f96599f98d82552bd913ce1b6135c7ce3b0a452a56e634e3dad6fa75559f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, release=754, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , architecture=x86_64, ceph=True) 2024-02-27T06:26:44.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:44 smithi150 bash[52646]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150 2024-02-27T06:26:45.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:44 smithi150 podman[52657]: 2024-02-27 06:26:44.86402298 +0000 UTC m=+0.686106313 container remove 976f96599f98d82552bd913ce1b6135c7ce3b0a452a56e634e3dad6fa75559f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150, io.openshift.expose-services=, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, architecture=x86_64, vcs-type=git, maintainer=Guillaume Abrioux , name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-27T06:26:45.285 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi150.service: Deactivated successfully. 2024-02-27T06:26:45.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 systemd[1]: Stopped Ceph mon.smithi150 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:26:45.286 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi150.service: Consumed 7.133s CPU time. 2024-02-27T06:26:45.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 164462 11.87 MB/sec execute 203 sec latency 40.505 ms 2024-02-27T06:26:45.540 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 systemd[1]: Starting Ceph mon.smithi150 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:26:45.825 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 podman[52785]: 2024-02-27 06:26:45.439540141 +0000 UTC m=+0.024911087 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:26:45.825 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 podman[52785]: 2024-02-27 06:26:45.573760559 +0000 UTC m=+0.159131486 container create 61da9fd9229099cc2527c8470cb148ff1ef6cf0019eb5794d5fa4d08ef2194a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:26:45.826 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 podman[52785]: 2024-02-27 06:26:45.818808042 +0000 UTC m=+0.404178980 container init 61da9fd9229099cc2527c8470cb148ff1ef6cf0019eb5794d5fa4d08ef2194a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux ) 2024-02-27T06:26:46.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 podman[52785]: 2024-02-27 06:26:45.823300057 +0000 UTC m=+0.408670988 container start 61da9fd9229099cc2527c8470cb148ff1ef6cf0019eb5794d5fa4d08ef2194a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-02-27T06:26:46.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: set uid:gid to 167:167 (ceph:ceph) 2024-02-27T06:26:46.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev), process ceph-mon, pid 2 2024-02-27T06:26:46.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: pidfile_write: ignore empty --pid-file 2024-02-27T06:26:46.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: load: jerasure load: lrc 2024-02-27T06:26:46.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: RocksDB version: 7.9.2 2024-02-27T06:26:46.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Git sha 0 2024-02-27T06:26:46.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Compile date 2024-02-26 21:43:15 2024-02-27T06:26:46.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: DB SUMMARY 2024-02-27T06:26:46.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: DB Session ID: S4NCEY18OH8GB0872NXQ 2024-02-27T06:26:46.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: CURRENT file: CURRENT 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: IDENTITY file: IDENTITY 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: MANIFEST file: MANIFEST-000009 size: 903 Bytes 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi150/store.db dir, Total Num: 1, files: 000024.sst 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi150/store.db: 000022.log size: 6587765 ; 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.error_if_exists: 0 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.create_if_missing: 0 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.paranoid_checks: 1 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.env: 0x563a5d3e0b20 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.fs: PosixFileSystem 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.info_log: 0x563a5eea7780 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_file_opening_threads: 16 2024-02-27T06:26:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.statistics: (nil) 2024-02-27T06:26:46.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.use_fsync: 0 2024-02-27T06:26:46.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_log_file_size: 0 2024-02-27T06:26:46.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-27T06:26:46.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-27T06:26:46.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.keep_log_file_num: 1000 2024-02-27T06:26:46.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.recycle_log_file_num: 0 2024-02-27T06:26:46.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.allow_fallocate: 1 2024-02-27T06:26:46.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.allow_mmap_reads: 0 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.allow_mmap_writes: 0 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.use_direct_reads: 0 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.create_missing_column_families: 0 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.db_log_dir: 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.wal_dir: 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.advise_random_on_open: 1 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.db_write_buffer_size: 0 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.write_buffer_manager: 0x563a5ee7b400 2024-02-27T06:26:46.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.rate_limiter: (nil) 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.wal_recovery_mode: 2 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.enable_thread_tracking: 0 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.enable_pipelined_write: 0 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.unordered_write: 0 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.row_cache: None 2024-02-27T06:26:46.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.wal_filter: None 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.allow_ingest_behind: 0 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.two_write_queues: 0 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.manual_wal_flush: 0 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.wal_compression: 0 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.atomic_flush: 0 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.log_readahead_size: 0 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.best_efforts_recovery: 0 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-27T06:26:46.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.allow_data_in_errors: 0 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.db_host_id: __hostname__ 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_background_jobs: 2 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_background_compactions: -1 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_subcompactions: 1 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_total_wal_size: 0 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-27T06:26:46.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-27T06:26:46.117 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_open_files: -1 2024-02-27T06:26:46.117 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bytes_per_sync: 0 2024-02-27T06:26:46.117 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-27T06:26:46.117 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-27T06:26:46.117 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_readahead_size: 0 2024-02-27T06:26:46.117 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_background_flushes: -1 2024-02-27T06:26:46.117 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Compression algorithms supported: 2024-02-27T06:26:46.117 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-27T06:26:46.117 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: kZSTD supported: 0 2024-02-27T06:26:46.117 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: kXpressCompression supported: 0 2024-02-27T06:26:46.117 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: kLZ4HCCompression supported: 1 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: kZlibCompression supported: 1 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: kSnappyCompression supported: 1 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: kLZ4Compression supported: 1 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: kBZip2Compression supported: 0 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi150/store.db/MANIFEST-000009 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.merge_operator: 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_filter: None 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_filter_factory: None 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.sst_partitioner_factory: None 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-27T06:26:46.118 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563a5eea78a0) 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: cache_index_and_filter_blocks: 1 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: pin_top_level_index_and_filter: 1 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: index_type: 0 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: data_block_index_type: 0 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: index_shortening: 1 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: checksum: 4 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: no_block_cache: 0 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_cache: 0x563a5eeab090 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_cache_name: BinnedLRUCache 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_cache_options: 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: capacity : 536870912 2024-02-27T06:26:46.119 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: num_shard_bits : 4 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: strict_capacity_limit : 0 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: high_pri_pool_ratio: 0.000 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_cache_compressed: (nil) 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: persistent_cache: (nil) 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_size: 4096 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_size_deviation: 10 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_restart_interval: 16 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: index_block_restart_interval: 1 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: metadata_block_size: 4096 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: partition_filters: 0 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: use_delta_encoding: 1 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: filter_policy: bloomfilter 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: whole_key_filtering: 1 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: verify_compression: 0 2024-02-27T06:26:46.120 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: read_amp_bytes_per_bit: 0 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: format_version: 5 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: enable_index_compression: 1 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: block_align: 0 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: max_auto_readahead_size: 262144 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: prepopulate_block_cache: 0 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: initial_auto_readahead_size: 8192 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: num_file_reads_for_auto_readahead: 2 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.write_buffer_size: 33554432 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_write_buffer_number: 2 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compression: NoCompression 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bottommost_compression: Disabled 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.prefix_extractor: nullptr 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.num_levels: 7 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-27T06:26:46.121 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compression_opts.level: 32767 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compression_opts.strategy: 0 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-27T06:26:46.122 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compression_opts.enabled: false 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.target_file_size_base: 67108864 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-27T06:26:46.123 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.arena_block_size: 1048576 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.disable_auto_compactions: 0 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-27T06:26:46.124 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.table_properties_collectors: 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.inplace_update_support: 0 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.bloom_locality: 0 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.max_successive_merges: 0 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.paranoid_file_checks: 0 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.force_consistency_checks: 1 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.report_bg_io_stats: 0 2024-02-27T06:26:46.125 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.ttl: 2592000 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.enable_blob_files: false 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.min_blob_size: 0 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.blob_file_size: 268435456 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.blob_file_starting_level: 0 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi150/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12258, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-27T06:26:46.126 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-02-27T06:26:46.127 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 71c2abac-429d-44c2-9d4f-c067b5dabd78 2024-02-27T06:26:46.127 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709015205860894, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-02-27T06:26:46.127 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-02-27T06:26:46.127 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709015205880761, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 4002482, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12255, "largest_seqno": 13936, "table_properties": {"data_size": 3995040, "index_size": 4759, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1669, "raw_key_size": 15348, "raw_average_key_size": 23, "raw_value_size": 3980562, "raw_average_value_size": 5985, "num_data_blocks": 216, "num_entries": 665, "num_filter_entries": 665, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709015205, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "71c2abac-429d-44c2-9d4f-c067b5dabd78", "db_session_id": "S4NCEY18OH8GB0872NXQ", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-02-27T06:26:46.127 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709015205880873, "job": 1, "event": "recovery_finished"} 2024-02-27T06:26:46.127 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-02-27T06:26:46.127 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-27T06:26:46.127 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi150/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-27T06:26:46.127 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563a5ef9a000 2024-02-27T06:26:46.127 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: DB pointer 0x563a5ef86000 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ** DB Stats ** 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ** Compaction Stats [default] ** 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-27T06:26:46.128 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: L0 1/0 3.82 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 424.9 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: L6 1/0 6.64 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Sum 2/0 10.46 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 424.9 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 424.9 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ** Compaction Stats [default] ** 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 424.9 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-27T06:26:46.129 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Cumulative compaction: 0.00 GB write, 139.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Interval compaction: 0.00 GB write, 139.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Block cache BinnedLRUCache@0x563a5eeab090#2 capacity: 512.00 MB usage: 6.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: starting mon.smithi150 rank 1 at public addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] at bind addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi150 fsid 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: mon.smithi150@-1(???) e2 preinit fsid 3fa4c766-d537-11ee-95c3-87774f69a715 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: mon.smithi150@-1(???).mds e7 new map 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: mon.smithi150@-1(???).mds e7 print_map 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: e7 2024-02-27T06:26:46.130 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: legacy client fscid: 1 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Filesystem 'foofs' (1) 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: fs_name foofs 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: epoch 5 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: created 2024-02-27T06:20:44.948420+0000 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: modified 2024-02-27T06:20:50.974048+0000 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: tableserver 0 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: root 0 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: session_timeout 60 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: session_autoclose 300 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: max_file_size 1099511627776 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: max_xattr_size 65536 2024-02-27T06:26:46.131 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: required_client_features {} 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: last_failure 0 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: last_failure_osd_epoch 0 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: max_mds 1 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: in 0 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: up {0=14476} 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: failed 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: damaged 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: stopped 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: data_pools [3] 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: metadata_pool 2 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: inline_data disabled 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: balancer 2024-02-27T06:26:46.132 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: bal_rank_mask -1 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: standby_count_wanted 1 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: [mds.foofs.smithi131.mhuwwb{0:14476} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.131:6834/3029043434,v1:172.21.15.131:6835/3029043434] compat {c=[1],r=[1],i=[7ff]}] 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: Standby daemons: 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout: [mds.foofs.smithi150.vsnjhu{-1:24273} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.150:6832/3371494173,v1:172.21.15.150:6833/3371494173] compat {c=[1],r=[1],i=[7ff]}] 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: mon.smithi150@-1(???).osd e73 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: mon.smithi150@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: mon.smithi150@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: mon.smithi150@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 ceph-mon[52803]: mon.smithi150@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 bash[52785]: 61da9fd9229099cc2527c8470cb148ff1ef6cf0019eb5794d5fa4d08ef2194a7 2024-02-27T06:26:46.133 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:45 smithi150 systemd[1]: Started Ceph mon.smithi150 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:26:46.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 164870 11.86 MB/sec execute 204 sec latency 39.089 ms 2024-02-27T06:26:47.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:47 smithi150 ceph-mon[52803]: pgmap v11: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.9 MiB/s wr, 508 op/s 2024-02-27T06:26:47.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:26:47.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:26:47.395 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:47 smithi131 ceph-mon[66259]: pgmap v11: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 24 KiB/s rd, 4.9 MiB/s wr, 508 op/s 2024-02-27T06:26:47.395 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:26:47.395 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:26:47.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 165547 11.89 MB/sec execute 205 sec latency 35.693 ms 2024-02-27T06:26:48.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: pgmap v13: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.7 MiB/s wr, 467 op/s 2024-02-27T06:26:48.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:26:48.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:26:48.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: mon.smithi131 calling monitor election 2024-02-27T06:26:48.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: mon.smithi150 calling monitor election 2024-02-27T06:26:48.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: mon.smithi131 is new leader, mons smithi131,smithi150 in quorum (ranks 0,1) 2024-02-27T06:26:48.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: monmap e3: 2 mons at {smithi131=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],smithi150=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T06:26:48.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 1 up:standby 2024-02-27T06:26:48.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: osdmap e73: 8 total, 8 up, 8 in 2024-02-27T06:26:48.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: mgrmap e41: smithi131.frtgqo(active, since 20s), standbys: smithi150.vvgbyr 2024-02-27T06:26:48.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: overall HEALTH_OK 2024-02-27T06:26:48.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:48.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:48.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:26:48.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 166064 11.91 MB/sec execute 206 sec latency 49.131 ms 2024-02-27T06:26:48.589 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: pgmap v13: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.7 MiB/s wr, 467 op/s 2024-02-27T06:26:48.589 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-02-27T06:26:48.590 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mon metadata", "id": "smithi150"}]: dispatch 2024-02-27T06:26:48.590 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: mon.smithi131 calling monitor election 2024-02-27T06:26:48.590 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: mon.smithi150 calling monitor election 2024-02-27T06:26:48.590 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: mon.smithi131 is new leader, mons smithi131,smithi150 in quorum (ranks 0,1) 2024-02-27T06:26:48.590 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: monmap e3: 2 mons at {smithi131=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],smithi150=[v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-27T06:26:48.590 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 1 up:standby 2024-02-27T06:26:48.590 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: osdmap e73: 8 total, 8 up, 8 in 2024-02-27T06:26:48.590 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: mgrmap e41: smithi131.frtgqo(active, since 20s), standbys: smithi150.vvgbyr 2024-02-27T06:26:48.590 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: overall HEALTH_OK 2024-02-27T06:26:48.590 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:48.590 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:48.590 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:26:49.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 166517 11.92 MB/sec execute 207 sec latency 40.522 ms 2024-02-27T06:26:50.186 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:50 smithi131 ceph-mon[66259]: pgmap v14: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 738 op/s 2024-02-27T06:26:50.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 166997 11.94 MB/sec execute 208 sec latency 37.265 ms 2024-02-27T06:26:50.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:50 smithi150 ceph-mon[52803]: pgmap v14: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 738 op/s 2024-02-27T06:26:51.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 167297 11.92 MB/sec execute 209 sec latency 44.556 ms 2024-02-27T06:26:52.224 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:52 smithi131 ceph-mon[66259]: pgmap v15: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.8 MiB/s wr, 470 op/s 2024-02-27T06:26:52.224 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:52 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:52.224 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:52 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:52.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 167592 11.91 MB/sec execute 210 sec latency 57.493 ms 2024-02-27T06:26:52.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:52 smithi150 ceph-mon[52803]: pgmap v15: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.8 MiB/s wr, 470 op/s 2024-02-27T06:26:52.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:52 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:52.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:52 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:53.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 167858 11.86 MB/sec execute 211 sec latency 76.246 ms 2024-02-27T06:26:54.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:54 smithi150 ceph-mon[52803]: pgmap v16: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.8 MiB/s wr, 470 op/s 2024-02-27T06:26:54.293 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:54.294 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:54.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:54 smithi131 ceph-mon[66259]: pgmap v16: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.8 MiB/s wr, 470 op/s 2024-02-27T06:26:54.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:54.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:54.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 168136 11.83 MB/sec execute 212 sec latency 55.608 ms 2024-02-27T06:26:55.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 168408 11.81 MB/sec execute 213 sec latency 41.826 ms 2024-02-27T06:26:56.186 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:56 smithi131 ceph-mon[66259]: pgmap v17: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.6 MiB/s wr, 727 op/s 2024-02-27T06:26:56.512 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:56 smithi150 ceph-mon[52803]: pgmap v17: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.6 MiB/s wr, 727 op/s 2024-02-27T06:26:56.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 168787 11.81 MB/sec execute 214 sec latency 64.826 ms 2024-02-27T06:26:57.226 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:26:57.452 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:26:57.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 169188 11.79 MB/sec execute 215 sec latency 51.451 ms 2024-02-27T06:26:58.257 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:58 smithi131 ceph-mon[66259]: pgmap v18: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.4 MiB/s wr, 527 op/s 2024-02-27T06:26:58.257 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:58.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:58.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:26:58.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:26:58.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:58.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:58.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:26:58.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T06:26:58.258 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:26:58.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 169496 11.79 MB/sec execute 216 sec latency 58.543 ms 2024-02-27T06:26:58.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:58 smithi150 ceph-mon[52803]: pgmap v18: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.4 MiB/s wr, 527 op/s 2024-02-27T06:26:58.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:58.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:58.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:26:58.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:26:58.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:58.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:26:58.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:26:58.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T06:26:58.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:26:59.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:59 smithi131 ceph-mon[66259]: Reconfiguring mon.smithi131 (monmap changed)... 2024-02-27T06:26:59.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:26:59 smithi131 ceph-mon[66259]: Reconfiguring daemon mon.smithi131 on smithi131 2024-02-27T06:26:59.524 INFO:teuthology.orchestra.run.smithi131.stdout: 5 169784 11.77 MB/sec execute 217 sec latency 93.675 ms 2024-02-27T06:26:59.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:59 smithi150 ceph-mon[52803]: Reconfiguring mon.smithi131 (monmap changed)... 2024-02-27T06:26:59.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:26:59 smithi150 ceph-mon[52803]: Reconfiguring daemon mon.smithi131 on smithi131 2024-02-27T06:27:00.308 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:00 smithi131 ceph-mon[66259]: pgmap v19: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 724 op/s 2024-02-27T06:27:00.528 INFO:teuthology.orchestra.run.smithi131.stdout: 5 170023 11.76 MB/sec execute 218 sec latency 52.838 ms 2024-02-27T06:27:00.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:00 smithi150 ceph-mon[52803]: pgmap v19: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 724 op/s 2024-02-27T06:27:00.807 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:27:01.528 INFO:teuthology.orchestra.run.smithi131.stdout: 5 170301 11.74 MB/sec execute 219 sec latency 55.377 ms 2024-02-27T06:27:01.865 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:01 smithi131 ceph-mon[66259]: pgmap v20: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.6 MiB/s wr, 453 op/s 2024-02-27T06:27:01.865 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:01.865 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:01.865 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:01 smithi131 ceph-mon[66259]: Reconfiguring mgr.smithi131.frtgqo (monmap changed)... 2024-02-27T06:27:01.865 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi131.frtgqo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:27:01.865 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:27:01.865 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:01.865 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:01 smithi131 ceph-mon[66259]: Reconfiguring daemon mgr.smithi131.frtgqo on smithi131 2024-02-27T06:27:01.865 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:01 smithi131 ceph-mon[66259]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:01.873 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:27:01.873 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (3m) 25s ago 12m 23.8M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:27:01.873 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (11m) 25s ago 11m 6405k - 17.2.0 e1d6a67b021e 5f66626dee40 2024-02-27T06:27:01.873 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (11m) 10s ago 11m 6405k - 17.2.0 e1d6a67b021e e37d494b0a36 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (2m) 25s ago 11m 52.2M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (6m) 25s ago 6m 38.3M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (6m) 10s ago 6m 13.9M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (2m) 25s ago 13m 476M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (68s) 10s ago 11m 447M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (47s) 25s ago 13m 30.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (16s) 10s ago 11m 23.6M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (5m) 25s ago 5m 262M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (5m) 10s ago 5m 81.4M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (2m) 25s ago 11m 10.0M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (2m) 10s ago 11m 18.5M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (10m) 25s ago 10m 562M 1936M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (10m) 25s ago 10m 487M 1936M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (9m) 25s ago 9m 362M 1936M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (9m) 25s ago 9m 488M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:27:01.874 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (8m) 10s ago 8m 527M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:27:01.875 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (8m) 10s ago 8m 355M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:27:01.875 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (8m) 10s ago 8m 543M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:27:01.875 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (7m) 10s ago 7m 447M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:27:01.875 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (75s) 25s ago 10m 52.9M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:27:02.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:01 smithi150 ceph-mon[52803]: pgmap v20: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.6 MiB/s wr, 453 op/s 2024-02-27T06:27:02.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:02.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:02.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:01 smithi150 ceph-mon[52803]: Reconfiguring mgr.smithi131.frtgqo (monmap changed)... 2024-02-27T06:27:02.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi131.frtgqo", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:27:02.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:27:02.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:02.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:01 smithi150 ceph-mon[52803]: Reconfiguring daemon mgr.smithi131.frtgqo on smithi131 2024-02-27T06:27:02.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:01 smithi150 ceph-mon[52803]: from='client.34152 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:02.437 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:27:02.438 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:27:02.438 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:27:02.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:27:02.438 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:27:02.438 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:27:02.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:27:02.438 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:27:02.438 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T06:27:02.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:27:02.438 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:27:02.438 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:27:02.438 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:27:02.439 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:27:02.439 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-27T06:27:02.439 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 4 2024-02-27T06:27:02.439 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:27:02.439 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:27:02.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 170583 11.73 MB/sec execute 220 sec latency 49.287 ms 2024-02-27T06:27:02.873 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:02 smithi131 ceph-mon[66259]: from='client.44105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:02.873 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:02 smithi131 ceph-mon[66259]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:02.874 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:02 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/3434588008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:27:02.959 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:27:02.959 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:27:02.959 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:27:02.959 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:27:02.959 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:27:02.959 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:27:02.959 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr" 2024-02-27T06:27:02.959 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:27:02.959 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "4/23 daemons upgraded", 2024-02-27T06:27:02.959 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading mon daemons", 2024-02-27T06:27:02.959 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:27:02.959 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:27:03.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:02 smithi150 ceph-mon[52803]: from='client.44105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:03.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:02 smithi150 ceph-mon[52803]: from='client.44109 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:03.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:02 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/3434588008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:27:03.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 170827 11.70 MB/sec execute 221 sec latency 45.430 ms 2024-02-27T06:27:03.593 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:27:03.893 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:03 smithi131 ceph-mon[66259]: pgmap v21: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.6 MiB/s wr, 453 op/s 2024-02-27T06:27:03.894 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:03 smithi131 ceph-mon[66259]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:03.894 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:03 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:03.894 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:03 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:03.894 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:03 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:27:03.894 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:03 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:03.894 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:03 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/3457597870' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:27:04.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:03 smithi150 ceph-mon[52803]: pgmap v21: 97 pgs: 97 active+clean; 333 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.6 MiB/s wr, 453 op/s 2024-02-27T06:27:04.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:03 smithi150 ceph-mon[52803]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:04.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:03 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:04.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:03 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:04.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:03 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:27:04.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:03 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:04.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:03 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/3457597870' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:27:04.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 171143 11.70 MB/sec execute 222 sec latency 36.681 ms 2024-02-27T06:27:04.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:04 smithi131 ceph-mon[66259]: Reconfiguring crash.smithi131 (monmap changed)... 2024-02-27T06:27:04.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:04 smithi131 ceph-mon[66259]: Reconfiguring daemon crash.smithi131 on smithi131 2024-02-27T06:27:05.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:04 smithi150 ceph-mon[52803]: Reconfiguring crash.smithi131 (monmap changed)... 2024-02-27T06:27:05.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:04 smithi150 ceph-mon[52803]: Reconfiguring daemon crash.smithi131 on smithi131 2024-02-27T06:27:05.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 171487 11.66 MB/sec execute 223 sec latency 66.910 ms 2024-02-27T06:27:05.934 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:05 smithi131 ceph-mon[66259]: pgmap v22: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.5 MiB/s wr, 620 op/s 2024-02-27T06:27:06.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:05 smithi150 ceph-mon[52803]: pgmap v22: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.5 MiB/s wr, 620 op/s 2024-02-27T06:27:06.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 171738 11.63 MB/sec execute 224 sec latency 46.226 ms 2024-02-27T06:27:07.219 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:07 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:07.219 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:07 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:07.219 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:07 smithi131 ceph-mon[66259]: Reconfiguring osd.0 (monmap changed)... 2024-02-27T06:27:07.219 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:07 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T06:27:07.219 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:07 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:07.219 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:07 smithi131 ceph-mon[66259]: Reconfiguring daemon osd.0 on smithi131 2024-02-27T06:27:07.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:07 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:07.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:07 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:07.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:07 smithi150 ceph-mon[52803]: Reconfiguring osd.0 (monmap changed)... 2024-02-27T06:27:07.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:07 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T06:27:07.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:07 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:07.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:07 smithi150 ceph-mon[52803]: Reconfiguring daemon osd.0 on smithi131 2024-02-27T06:27:07.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 171975 11.60 MB/sec execute 225 sec latency 47.353 ms 2024-02-27T06:27:08.242 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:08 smithi131 ceph-mon[66259]: pgmap v23: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.6 MiB/s wr, 364 op/s 2024-02-27T06:27:08.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:08 smithi150 ceph-mon[52803]: pgmap v23: 97 pgs: 97 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.6 MiB/s wr, 364 op/s 2024-02-27T06:27:08.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 172289 11.61 MB/sec execute 226 sec latency 65.193 ms 2024-02-27T06:27:09.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 172650 11.58 MB/sec execute 227 sec latency 67.679 ms 2024-02-27T06:27:09.798 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:09 smithi131 ceph-mon[66259]: pgmap v24: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.4 MiB/s wr, 551 op/s 2024-02-27T06:27:09.799 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:09.799 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:09.799 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:09 smithi131 ceph-mon[66259]: Reconfiguring osd.1 (monmap changed)... 2024-02-27T06:27:09.799 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T06:27:09.799 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:09.799 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:09 smithi131 ceph-mon[66259]: Reconfiguring daemon osd.1 on smithi131 2024-02-27T06:27:09.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:09 smithi150 ceph-mon[52803]: pgmap v24: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.4 MiB/s wr, 551 op/s 2024-02-27T06:27:09.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:09.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:09.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:09 smithi150 ceph-mon[52803]: Reconfiguring osd.1 (monmap changed)... 2024-02-27T06:27:09.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T06:27:09.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:09.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:09 smithi150 ceph-mon[52803]: Reconfiguring daemon osd.1 on smithi131 2024-02-27T06:27:10.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 173029 11.59 MB/sec execute 228 sec latency 80.069 ms 2024-02-27T06:27:11.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 173436 11.60 MB/sec execute 229 sec latency 28.421 ms 2024-02-27T06:27:12.325 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:12 smithi131 ceph-mon[66259]: pgmap v25: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 354 op/s 2024-02-27T06:27:12.325 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:12 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:12.325 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:12 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:12.325 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:12 smithi131 ceph-mon[66259]: Reconfiguring osd.2 (monmap changed)... 2024-02-27T06:27:12.325 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:12 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T06:27:12.325 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:12 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:12.325 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:12 smithi131 ceph-mon[66259]: Reconfiguring daemon osd.2 on smithi131 2024-02-27T06:27:12.325 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:12 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:27:12.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:12 smithi150 ceph-mon[52803]: pgmap v25: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 354 op/s 2024-02-27T06:27:12.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:12 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:12.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:12 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:12.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:12 smithi150 ceph-mon[52803]: Reconfiguring osd.2 (monmap changed)... 2024-02-27T06:27:12.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:12 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T06:27:12.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:12 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:12.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:12 smithi150 ceph-mon[52803]: Reconfiguring daemon osd.2 on smithi131 2024-02-27T06:27:12.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:12 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:27:12.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 173812 11.59 MB/sec execute 230 sec latency 52.681 ms 2024-02-27T06:27:13.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 174135 11.59 MB/sec execute 231 sec latency 41.498 ms 2024-02-27T06:27:14.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:14 smithi131 ceph-mon[66259]: pgmap v26: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 354 op/s 2024-02-27T06:27:14.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:14.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:14.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T06:27:14.098 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:14.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:14 smithi150 ceph-mon[52803]: pgmap v26: 97 pgs: 97 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 354 op/s 2024-02-27T06:27:14.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:14.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:14.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T06:27:14.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:14.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 174380 11.56 MB/sec execute 232 sec latency 50.669 ms 2024-02-27T06:27:15.126 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:15 smithi131 ceph-mon[66259]: Reconfiguring osd.3 (monmap changed)... 2024-02-27T06:27:15.126 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:15 smithi131 ceph-mon[66259]: Reconfiguring daemon osd.3 on smithi131 2024-02-27T06:27:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:15 smithi150 ceph-mon[52803]: Reconfiguring osd.3 (monmap changed)... 2024-02-27T06:27:15.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:15 smithi150 ceph-mon[52803]: Reconfiguring daemon osd.3 on smithi131 2024-02-27T06:27:15.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 174697 11.56 MB/sec execute 233 sec latency 52.227 ms 2024-02-27T06:27:16.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:16 smithi131 ceph-mon[66259]: pgmap v27: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 562 op/s 2024-02-27T06:27:16.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:16 smithi150 ceph-mon[52803]: pgmap v27: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.8 MiB/s wr, 562 op/s 2024-02-27T06:27:16.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 175038 11.52 MB/sec execute 234 sec latency 53.925 ms 2024-02-27T06:27:17.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 175297 11.49 MB/sec execute 235 sec latency 45.969 ms 2024-02-27T06:27:17.693 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:17 smithi131 ceph-mon[66259]: pgmap v28: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 395 op/s 2024-02-27T06:27:17.694 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:17 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:17.694 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:17 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:17.694 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:17 smithi131 ceph-mon[66259]: Reconfiguring mds.foofs.smithi131.mhuwwb (monmap changed)... 2024-02-27T06:27:17.694 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:17 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi131.mhuwwb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-27T06:27:17.694 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:17 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:17.694 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:17 smithi131 ceph-mon[66259]: Reconfiguring daemon mds.foofs.smithi131.mhuwwb on smithi131 2024-02-27T06:27:17.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:17 smithi150 ceph-mon[52803]: pgmap v28: 97 pgs: 97 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 395 op/s 2024-02-27T06:27:17.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:17 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:17.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:17 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:17.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:17 smithi150 ceph-mon[52803]: Reconfiguring mds.foofs.smithi131.mhuwwb (monmap changed)... 2024-02-27T06:27:17.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:17 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi131.mhuwwb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-27T06:27:17.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:17 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:17.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:17 smithi150 ceph-mon[52803]: Reconfiguring daemon mds.foofs.smithi131.mhuwwb on smithi131 2024-02-27T06:27:18.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 175572 11.47 MB/sec execute 236 sec latency 39.946 ms 2024-02-27T06:27:19.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 175923 11.48 MB/sec execute 237 sec latency 54.533 ms 2024-02-27T06:27:19.732 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:19 smithi131 ceph-mon[66259]: pgmap v29: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.8 MiB/s wr, 589 op/s 2024-02-27T06:27:19.733 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:19.733 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:19.733 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:27:19.733 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:19.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:19 smithi150 ceph-mon[52803]: pgmap v29: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.8 MiB/s wr, 589 op/s 2024-02-27T06:27:19.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:19.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:19.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:27:19.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:20.523 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:20 smithi131 ceph-mon[66259]: Reconfiguring crash.smithi150 (monmap changed)... 2024-02-27T06:27:20.523 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:20 smithi131 ceph-mon[66259]: Reconfiguring daemon crash.smithi150 on smithi150 2024-02-27T06:27:20.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 176516 11.50 MB/sec execute 238 sec latency 64.139 ms 2024-02-27T06:27:20.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:20 smithi150 ceph-mon[52803]: Reconfiguring crash.smithi150 (monmap changed)... 2024-02-27T06:27:20.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:20 smithi150 ceph-mon[52803]: Reconfiguring daemon crash.smithi150 on smithi150 2024-02-27T06:27:21.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 177073 11.52 MB/sec execute 239 sec latency 35.772 ms 2024-02-27T06:27:21.531 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:21 smithi131 ceph-mon[66259]: pgmap v30: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 402 op/s 2024-02-27T06:27:21.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:21 smithi150 ceph-mon[52803]: pgmap v30: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 402 op/s 2024-02-27T06:27:22.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 177578 11.54 MB/sec execute 240 sec latency 25.840 ms 2024-02-27T06:27:23.058 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:23.059 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:23.059 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:22 smithi131 ceph-mon[66259]: Reconfiguring mgr.smithi150.vvgbyr (monmap changed)... 2024-02-27T06:27:23.059 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vvgbyr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:27:23.059 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:27:23.059 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:23.059 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:22 smithi131 ceph-mon[66259]: Reconfiguring daemon mgr.smithi150.vvgbyr on smithi150 2024-02-27T06:27:23.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:23.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:23.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:22 smithi150 ceph-mon[52803]: Reconfiguring mgr.smithi150.vvgbyr (monmap changed)... 2024-02-27T06:27:23.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi150.vvgbyr", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-27T06:27:23.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-27T06:27:23.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:23.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:22 smithi150 ceph-mon[52803]: Reconfiguring daemon mgr.smithi150.vvgbyr on smithi150 2024-02-27T06:27:23.529 INFO:teuthology.orchestra.run.smithi131.stdout: 5 178027 11.54 MB/sec execute 241 sec latency 33.353 ms 2024-02-27T06:27:24.063 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:23 smithi131 ceph-mon[66259]: pgmap v31: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 402 op/s 2024-02-27T06:27:24.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:23 smithi150 ceph-mon[52803]: pgmap v31: 97 pgs: 97 active+clean; 334 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 402 op/s 2024-02-27T06:27:24.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 178547 11.53 MB/sec execute 242 sec latency 49.838 ms 2024-02-27T06:27:25.074 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:24 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:25.074 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:24 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:25.075 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:24 smithi131 ceph-mon[66259]: Reconfiguring mon.smithi150 (monmap changed)... 2024-02-27T06:27:25.075 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:24 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:27:25.075 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:24 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T06:27:25.075 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:24 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:25.075 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:24 smithi131 ceph-mon[66259]: Reconfiguring daemon mon.smithi150 on smithi150 2024-02-27T06:27:25.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:24 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:25.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:24 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:25.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:24 smithi150 ceph-mon[52803]: Reconfiguring mon.smithi150 (monmap changed)... 2024-02-27T06:27:25.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:24 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-27T06:27:25.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:24 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-27T06:27:25.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:24 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:25.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:24 smithi150 ceph-mon[52803]: Reconfiguring daemon mon.smithi150 on smithi150 2024-02-27T06:27:25.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 178979 11.53 MB/sec execute 243 sec latency 38.020 ms 2024-02-27T06:27:26.086 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:26 smithi131 ceph-mon[66259]: pgmap v32: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.8 MiB/s wr, 669 op/s 2024-02-27T06:27:26.086 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:26.086 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:26.086 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T06:27:26.086 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:26.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:26 smithi150 ceph-mon[52803]: pgmap v32: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 6.8 MiB/s wr, 669 op/s 2024-02-27T06:27:26.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:26.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:26.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T06:27:26.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:26.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 179435 11.54 MB/sec execute 244 sec latency 47.482 ms 2024-02-27T06:27:27.093 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:27 smithi131 ceph-mon[66259]: Reconfiguring osd.4 (monmap changed)... 2024-02-27T06:27:27.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:27 smithi131 ceph-mon[66259]: Reconfiguring daemon osd.4 on smithi150 2024-02-27T06:27:27.094 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:27:27.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:27 smithi150 ceph-mon[52803]: Reconfiguring osd.4 (monmap changed)... 2024-02-27T06:27:27.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:27 smithi150 ceph-mon[52803]: Reconfiguring daemon osd.4 on smithi150 2024-02-27T06:27:27.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:27:27.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 180077 11.56 MB/sec execute 245 sec latency 19.508 ms 2024-02-27T06:27:28.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:28 smithi131 ceph-mon[66259]: pgmap v33: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 461 op/s 2024-02-27T06:27:28.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:28 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:28.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:28 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:28.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:28 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T06:27:28.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:28 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:28.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:28 smithi150 ceph-mon[52803]: pgmap v33: 97 pgs: 97 active+clean; 341 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 461 op/s 2024-02-27T06:27:28.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:28 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:28.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:28 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:28.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:28 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T06:27:28.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:28 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:28.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 180531 11.57 MB/sec execute 246 sec latency 36.708 ms 2024-02-27T06:27:29.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:29 smithi150 ceph-mon[52803]: Reconfiguring osd.5 (monmap changed)... 2024-02-27T06:27:29.108 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:29 smithi150 ceph-mon[52803]: Reconfiguring daemon osd.5 on smithi150 2024-02-27T06:27:29.129 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:29 smithi131 ceph-mon[66259]: Reconfiguring osd.5 (monmap changed)... 2024-02-27T06:27:29.130 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:29 smithi131 ceph-mon[66259]: Reconfiguring daemon osd.5 on smithi150 2024-02-27T06:27:29.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 180974 11.57 MB/sec execute 247 sec latency 26.767 ms 2024-02-27T06:27:30.140 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:30 smithi131 ceph-mon[66259]: pgmap v34: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 765 op/s 2024-02-27T06:27:30.140 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:30.140 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:30.140 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T06:27:30.140 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:30.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:30 smithi150 ceph-mon[52803]: pgmap v34: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 765 op/s 2024-02-27T06:27:30.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:30.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:30.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T06:27:30.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:30.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 181333 11.58 MB/sec execute 248 sec latency 34.650 ms 2024-02-27T06:27:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:31 smithi131 ceph-mon[66259]: Reconfiguring osd.6 (monmap changed)... 2024-02-27T06:27:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:31 smithi131 ceph-mon[66259]: Reconfiguring daemon osd.6 on smithi150 2024-02-27T06:27:31.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:31 smithi150 ceph-mon[52803]: Reconfiguring osd.6 (monmap changed)... 2024-02-27T06:27:31.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:31 smithi150 ceph-mon[52803]: Reconfiguring daemon osd.6 on smithi150 2024-02-27T06:27:31.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 181812 11.58 MB/sec execute 249 sec latency 31.816 ms 2024-02-27T06:27:32.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:32 smithi131 ceph-mon[66259]: pgmap v35: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.7 MiB/s wr, 570 op/s 2024-02-27T06:27:32.171 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:32 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:32.171 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:32 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:32.171 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:32 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T06:27:32.171 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:32 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:32.291 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:32 smithi150 ceph-mon[52803]: pgmap v35: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.7 MiB/s wr, 570 op/s 2024-02-27T06:27:32.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:32 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:32.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:32 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:32.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:32 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T06:27:32.292 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:32 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:32.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 182399 11.57 MB/sec execute 250 sec latency 32.069 ms 2024-02-27T06:27:33.185 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:33 smithi131 ceph-mon[66259]: Reconfiguring osd.7 (monmap changed)... 2024-02-27T06:27:33.185 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:33 smithi131 ceph-mon[66259]: Reconfiguring daemon osd.7 on smithi150 2024-02-27T06:27:33.319 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:33 smithi150 ceph-mon[52803]: Reconfiguring osd.7 (monmap changed)... 2024-02-27T06:27:33.319 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:33 smithi150 ceph-mon[52803]: Reconfiguring daemon osd.7 on smithi150 2024-02-27T06:27:33.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 182947 11.59 MB/sec execute 251 sec latency 29.756 ms 2024-02-27T06:27:34.132 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:27:34.191 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:34 smithi131 ceph-mon[66259]: pgmap v36: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.7 MiB/s wr, 570 op/s 2024-02-27T06:27:34.191 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:34 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:34.191 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:34 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:34.191 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:34 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi150.vsnjhu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-27T06:27:34.191 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:34 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:34.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:34 smithi150 ceph-mon[52803]: pgmap v36: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.7 MiB/s wr, 570 op/s 2024-02-27T06:27:34.311 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:34 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:34.312 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:34 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:34.312 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:34 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi150.vsnjhu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-27T06:27:34.312 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:34 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:34.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 183615 11.62 MB/sec execute 252 sec latency 32.130 ms 2024-02-27T06:27:35.200 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:35 smithi150 ceph-mon[52803]: Reconfiguring mds.foofs.smithi150.vsnjhu (monmap changed)... 2024-02-27T06:27:35.200 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:35 smithi150 ceph-mon[52803]: Reconfiguring daemon mds.foofs.smithi150.vsnjhu on smithi150 2024-02-27T06:27:35.200 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:35 smithi150 ceph-mon[52803]: from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:35.201 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:35 smithi131 ceph-mon[66259]: Reconfiguring mds.foofs.smithi150.vsnjhu (monmap changed)... 2024-02-27T06:27:35.201 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:35 smithi131 ceph-mon[66259]: Reconfiguring daemon mds.foofs.smithi150.vsnjhu on smithi150 2024-02-27T06:27:35.202 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:35 smithi131 ceph-mon[66259]: from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:35.221 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:27:35.221 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (3m) 58s ago 12m 23.8M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (12m) 58s ago 12m 6405k - 17.2.0 e1d6a67b021e 5f66626dee40 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (12m) 43s ago 12m 6405k - 17.2.0 e1d6a67b021e e37d494b0a36 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (3m) 58s ago 12m 52.2M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (6m) 58s ago 6m 38.3M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (6m) 43s ago 6m 13.9M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (2m) 58s ago 13m 476M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (101s) 43s ago 11m 447M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (80s) 58s ago 13m 30.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (49s) 43s ago 11m 23.6M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (6m) 58s ago 6m 262M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (6m) 43s ago 6m 81.4M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (3m) 58s ago 12m 10.0M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (2m) 43s ago 11m 18.5M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (11m) 58s ago 11m 562M 1936M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:27:35.222 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (10m) 58s ago 10m 487M 1936M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:27:35.223 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (10m) 58s ago 10m 362M 1936M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:27:35.223 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (9m) 58s ago 9m 488M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:27:35.223 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (9m) 43s ago 9m 527M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:27:35.223 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (9m) 43s ago 9m 355M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:27:35.223 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (8m) 43s ago 8m 543M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:27:35.223 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (8m) 43s ago 8m 447M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:27:35.223 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (109s) 58s ago 11m 52.9M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:27:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 184152 11.63 MB/sec execute 253 sec latency 55.432 ms 2024-02-27T06:27:35.870 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:27:35.870 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:27:35.870 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:27:35.870 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:27:35.870 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 4 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:27:35.871 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:27:36.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: pgmap v37: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 867 op/s 2024-02-27T06:27:36.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:36.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='client.44131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:36.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:36.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:36.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:27:36.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:27:36.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:27:36.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:36.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi131"}]: dispatch 2024-02-27T06:27:36.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi131"}]': finished 2024-02-27T06:27:36.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi150"}]: dispatch 2024-02-27T06:27:36.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi150"}]': finished 2024-02-27T06:27:36.231 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:36 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/3737507851' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:27:36.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: pgmap v37: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 8.7 MiB/s wr, 867 op/s 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='client.44131 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi131"}]: dispatch 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi131"}]': finished 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi150"}]: dispatch 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi150"}]': finished 2024-02-27T06:27:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:36 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/3737507851' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:27:36.396 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:27:36.396 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:27:36.396 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:27:36.396 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:27:36.396 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:27:36.397 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:27:36.397 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr" 2024-02-27T06:27:36.397 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:27:36.397 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "4/23 daemons upgraded", 2024-02-27T06:27:36.397 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading crash daemons", 2024-02-27T06:27:36.397 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:27:36.397 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:27:36.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 184390 11.61 MB/sec execute 254 sec latency 62.773 ms 2024-02-27T06:27:37.018 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:27:37.251 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:37 smithi131 ceph-mon[66259]: Upgrade: Setting container_image for all mon 2024-02-27T06:27:37.252 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:37 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/3808683694' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:27:37.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:37 smithi150 ceph-mon[52803]: Upgrade: Setting container_image for all mon 2024-02-27T06:27:37.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:37 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/3808683694' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:27:37.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 184627 11.60 MB/sec execute 255 sec latency 48.207 ms 2024-02-27T06:27:38.267 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:38 smithi131 ceph-mon[66259]: from='client.44135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:38.267 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:38 smithi131 ceph-mon[66259]: pgmap v38: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.0 MiB/s wr, 600 op/s 2024-02-27T06:27:38.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:38 smithi150 ceph-mon[52803]: from='client.44135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:27:38.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:38 smithi150 ceph-mon[52803]: pgmap v38: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.0 MiB/s wr, 600 op/s 2024-02-27T06:27:38.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 184849 11.59 MB/sec execute 256 sec latency 45.237 ms 2024-02-27T06:27:39.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 185174 11.58 MB/sec execute 257 sec latency 36.851 ms 2024-02-27T06:27:39.546 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:39 smithi131 ceph-mon[66259]: Upgrade: Updating crash.smithi131 (1/2) 2024-02-27T06:27:39.546 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:39 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:39.546 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:39 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:27:39.546 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:39 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:39.546 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:39 smithi131 ceph-mon[66259]: Deploying daemon crash.smithi131 on smithi131 2024-02-27T06:27:39.546 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:39 smithi131 ceph-mon[66259]: pgmap v39: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 863 op/s 2024-02-27T06:27:39.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:39 smithi150 ceph-mon[52803]: Upgrade: Updating crash.smithi131 (1/2) 2024-02-27T06:27:39.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:39 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:39.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:39 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:27:39.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:39 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:39.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:39 smithi150 ceph-mon[52803]: Deploying daemon crash.smithi131 on smithi131 2024-02-27T06:27:39.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:39 smithi150 ceph-mon[52803]: pgmap v39: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 863 op/s 2024-02-27T06:27:40.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 185698 11.56 MB/sec execute 258 sec latency 31.889 ms 2024-02-27T06:27:41.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 186052 11.54 MB/sec execute 259 sec latency 101.946 ms 2024-02-27T06:27:41.581 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:41 smithi131 ceph-mon[66259]: pgmap v40: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.8 MiB/s wr, 559 op/s 2024-02-27T06:27:41.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:41 smithi150 ceph-mon[52803]: pgmap v40: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.8 MiB/s wr, 559 op/s 2024-02-27T06:27:42.531 INFO:teuthology.orchestra.run.smithi131.stdout: 5 186434 11.56 MB/sec execute 260 sec latency 80.095 ms 2024-02-27T06:27:42.587 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:27:42.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:27:43.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 186822 11.54 MB/sec execute 261 sec latency 44.339 ms 2024-02-27T06:27:43.611 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:43 smithi131 ceph-mon[66259]: pgmap v41: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.8 MiB/s wr, 559 op/s 2024-02-27T06:27:43.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:43 smithi150 ceph-mon[52803]: pgmap v41: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.8 MiB/s wr, 559 op/s 2024-02-27T06:27:44.531 INFO:teuthology.orchestra.run.smithi131.stdout: 5 187144 11.54 MB/sec execute 262 sec latency 64.089 ms 2024-02-27T06:27:45.531 INFO:teuthology.orchestra.run.smithi131.stdout: 5 187462 11.53 MB/sec execute 263 sec latency 55.861 ms 2024-02-27T06:27:46.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:46 smithi131 ceph-mon[66259]: pgmap v42: 97 pgs: 97 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 788 op/s 2024-02-27T06:27:46.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:46 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:46.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:46 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:46.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:46 smithi150 ceph-mon[52803]: pgmap v42: 97 pgs: 97 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 788 op/s 2024-02-27T06:27:46.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:46 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:46.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:46 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:46.530 INFO:teuthology.orchestra.run.smithi131.stdout: 5 187992 11.54 MB/sec execute 264 sec latency 27.737 ms 2024-02-27T06:27:47.531 INFO:teuthology.orchestra.run.smithi131.stdout: 5 188498 11.56 MB/sec execute 265 sec latency 45.026 ms 2024-02-27T06:27:47.883 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:47 smithi150 ceph-mon[52803]: pgmap v43: 97 pgs: 97 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 492 op/s 2024-02-27T06:27:47.883 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:47 smithi150 ceph-mon[52803]: Upgrade: Updating crash.smithi150 (2/2) 2024-02-27T06:27:47.883 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:47.884 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:27:47.884 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:47.884 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:47 smithi150 ceph-mon[52803]: Deploying daemon crash.smithi150 on smithi150 2024-02-27T06:27:47.937 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:47 smithi131 ceph-mon[66259]: pgmap v43: 97 pgs: 97 active+clean; 330 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.9 MiB/s wr, 492 op/s 2024-02-27T06:27:47.937 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:47 smithi131 ceph-mon[66259]: Upgrade: Updating crash.smithi150 (2/2) 2024-02-27T06:27:47.937 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:47.937 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi150", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-27T06:27:47.937 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:27:47.937 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:47 smithi131 ceph-mon[66259]: Deploying daemon crash.smithi150 on smithi150 2024-02-27T06:27:48.531 INFO:teuthology.orchestra.run.smithi131.stdout: 5 189093 11.57 MB/sec execute 266 sec latency 34.857 ms 2024-02-27T06:27:49.531 INFO:teuthology.orchestra.run.smithi131.stdout: 5 189597 11.56 MB/sec execute 267 sec latency 32.781 ms 2024-02-27T06:27:49.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:49 smithi131 ceph-mon[66259]: pgmap v44: 97 pgs: 97 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.7 MiB/s wr, 744 op/s 2024-02-27T06:27:49.993 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:49 smithi150 ceph-mon[52803]: pgmap v44: 97 pgs: 97 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.7 MiB/s wr, 744 op/s 2024-02-27T06:27:50.531 INFO:teuthology.orchestra.run.smithi131.stdout: 5 190151 11.58 MB/sec execute 268 sec latency 46.455 ms 2024-02-27T06:27:51.531 INFO:teuthology.orchestra.run.smithi131.stdout: 5 190683 11.60 MB/sec execute 269 sec latency 65.903 ms 2024-02-27T06:27:51.933 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:51 smithi131 ceph-mon[66259]: pgmap v45: 97 pgs: 97 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.0 MiB/s wr, 481 op/s 2024-02-27T06:27:52.071 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:51 smithi150 ceph-mon[52803]: pgmap v45: 97 pgs: 97 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.0 MiB/s wr, 481 op/s 2024-02-27T06:27:52.531 INFO:teuthology.orchestra.run.smithi131.stdout: 5 190934 11.58 MB/sec execute 270 sec latency 78.257 ms 2024-02-27T06:27:53.479 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:53 smithi131 ceph-mon[66259]: pgmap v46: 97 pgs: 97 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.0 MiB/s wr, 481 op/s 2024-02-27T06:27:53.479 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:53.479 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:53.479 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:27:53.531 INFO:teuthology.orchestra.run.smithi131.stdout: 5 191198 11.56 MB/sec execute 271 sec latency 63.023 ms 2024-02-27T06:27:53.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:53 smithi150 ceph-mon[52803]: pgmap v46: 97 pgs: 97 active+clean; 338 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 5.0 MiB/s wr, 481 op/s 2024-02-27T06:27:53.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:53.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:53.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:27:54.531 INFO:teuthology.orchestra.run.smithi131.stdout: 5 191407 11.54 MB/sec execute 272 sec latency 69.077 ms 2024-02-27T06:27:55.531 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:55 smithi131 ceph-mon[66259]: pgmap v47: 97 pgs: 97 active+clean; 332 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 782 op/s 2024-02-27T06:27:55.531 INFO:teuthology.orchestra.run.smithi131.stdout: 5 191660 11.52 MB/sec execute 273 sec latency 73.194 ms 2024-02-27T06:27:55.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:55 smithi150 ceph-mon[52803]: pgmap v47: 97 pgs: 97 active+clean; 332 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.7 MiB/s wr, 782 op/s 2024-02-27T06:27:56.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 191896 11.52 MB/sec execute 274 sec latency 55.254 ms 2024-02-27T06:27:57.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 192176 11.50 MB/sec execute 275 sec latency 61.785 ms 2024-02-27T06:27:57.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:57.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:57.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:57 smithi131 ceph-mon[66259]: pgmap v48: 97 pgs: 97 active+clean; 332 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.6 MiB/s wr, 553 op/s 2024-02-27T06:27:57.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:27:57.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:57.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:57.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:57 smithi150 ceph-mon[52803]: pgmap v48: 97 pgs: 97 active+clean; 332 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.6 MiB/s wr, 553 op/s 2024-02-27T06:27:57.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:27:58.510 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:58 smithi150 ceph-mon[52803]: mgrmap e42: smithi131.frtgqo(active, since 91s), standbys: smithi150.vvgbyr 2024-02-27T06:27:58.511 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:58.511 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:58.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 192490 11.49 MB/sec execute 276 sec latency 83.658 ms 2024-02-27T06:27:58.587 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:58 smithi131 ceph-mon[66259]: mgrmap e42: smithi131.frtgqo(active, since 91s), standbys: smithi150.vvgbyr 2024-02-27T06:27:58.588 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:58.588 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:27:59.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 192782 11.46 MB/sec execute 277 sec latency 68.717 ms 2024-02-27T06:27:59.601 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:27:59 smithi131 ceph-mon[66259]: pgmap v49: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 711 op/s 2024-02-27T06:27:59.752 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:27:59 smithi150 ceph-mon[52803]: pgmap v49: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 711 op/s 2024-02-27T06:28:00.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 193085 11.44 MB/sec execute 278 sec latency 55.062 ms 2024-02-27T06:28:01.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 193535 11.46 MB/sec execute 279 sec latency 41.228 ms 2024-02-27T06:28:02.377 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:02 smithi131 ceph-mon[66259]: pgmap v50: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-02-27T06:28:02.377 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:02 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:02.377 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:02 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:02.377 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:02 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:02.377 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:02 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:02.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 193945 11.44 MB/sec execute 280 sec latency 47.542 ms 2024-02-27T06:28:02.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:02 smithi150 ceph-mon[52803]: pgmap v50: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-02-27T06:28:02.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:02 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:02.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:02 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:02.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:02 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:02.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:02 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:03.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 194286 11.45 MB/sec execute 281 sec latency 45.994 ms 2024-02-27T06:28:04.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:03 smithi131 ceph-mon[66259]: pgmap v51: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-02-27T06:28:04.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:03 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:04.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:03 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:04.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:03 smithi150 ceph-mon[52803]: pgmap v51: 97 pgs: 97 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-02-27T06:28:04.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:03 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:04.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:03 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:04.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 194590 11.43 MB/sec execute 282 sec latency 57.256 ms 2024-02-27T06:28:05.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 194911 11.43 MB/sec execute 283 sec latency 46.551 ms 2024-02-27T06:28:06.221 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:05 smithi131 ceph-mon[66259]: pgmap v52: 97 pgs: 97 active+clean; 332 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 669 op/s 2024-02-27T06:28:06.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:05 smithi150 ceph-mon[52803]: pgmap v52: 97 pgs: 97 active+clean; 332 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 669 op/s 2024-02-27T06:28:06.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 195224 11.41 MB/sec execute 284 sec latency 45.454 ms 2024-02-27T06:28:07.534 INFO:teuthology.orchestra.run.smithi131.stdout: 5 195529 11.41 MB/sec execute 285 sec latency 53.266 ms 2024-02-27T06:28:07.629 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:28:08.019 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:08 smithi131 ceph-mon[66259]: pgmap v53: 97 pgs: 97 active+clean; 332 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.7 MiB/s wr, 368 op/s 2024-02-27T06:28:08.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:08 smithi150 ceph-mon[52803]: pgmap v53: 97 pgs: 97 active+clean; 332 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.7 MiB/s wr, 368 op/s 2024-02-27T06:28:08.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 195886 11.42 MB/sec execute 286 sec latency 95.388 ms 2024-02-27T06:28:08.665 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:28:08.665 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (4m) 7s ago 13m 22.5M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:28:08.665 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (24s) 7s ago 13m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:28:08.665 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (16s) 12s ago 12m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:28:08.665 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (3m) 7s ago 12m 53.6M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:28:08.665 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (7m) 7s ago 7m 38.6M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:28:08.665 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (7m) 12s ago 7m 14.2M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (3m) 7s ago 14m 497M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (2m) 12s ago 12m 449M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (113s) 7s ago 14m 41.6M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (83s) 12s ago 12m 36.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (6m) 7s ago 6m 264M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (6m) 12s ago 6m 81.4M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (3m) 7s ago 12m 17.1M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (3m) 12s ago 12m 17.2M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (11m) 7s ago 11m 395M 1936M 17.2.0 e1d6a67b021e 09befbc8743b 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (11m) 7s ago 11m 582M 1936M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (10m) 7s ago 10m 470M 1936M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (10m) 7s ago 10m 588M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (9m) 12s ago 9m 557M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (9m) 12s ago 9m 436M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (9m) 12s ago 9m 347M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (8m) 12s ago 8m 542M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:28:08.666 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (2m) 7s ago 12m 65.6M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:28:09.042 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='client.34198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:09.042 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi131"}]: dispatch 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi131"}]': finished 2024-02-27T06:28:09.043 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi150"}]: dispatch 2024-02-27T06:28:09.044 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi150"}]': finished 2024-02-27T06:28:09.044 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:28:09.359 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-27T06:28:09.360 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 4 2024-02-27T06:28:09.360 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:28:09.360 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:28:09.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='client.34198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:09.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:09.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:09.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:09.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:28:09.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:28:09.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:09.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:09.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:28:09.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:09.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:09.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:09.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:09.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi131"}]: dispatch 2024-02-27T06:28:09.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi131"}]': finished 2024-02-27T06:28:09.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi150"}]: dispatch 2024-02-27T06:28:09.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi150"}]': finished 2024-02-27T06:28:09.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-27T06:28:09.534 INFO:teuthology.orchestra.run.smithi131.stdout: 5 196250 11.39 MB/sec execute 287 sec latency 72.552 ms 2024-02-27T06:28:09.858 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:28:09.858 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:28:09.858 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:28:09.858 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:28:09.858 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:28:09.858 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:28:09.858 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:28:09.858 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:28:09.859 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:28:09.859 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "6/23 daemons upgraded", 2024-02-27T06:28:09.859 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T06:28:09.859 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:28:09.859 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:28:10.046 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:10 smithi131 ceph-mon[66259]: pgmap v54: 97 pgs: 97 active+clean; 336 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.9 MiB/s wr, 567 op/s 2024-02-27T06:28:10.046 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:10 smithi131 ceph-mon[66259]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:10.046 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:10 smithi131 ceph-mon[66259]: Upgrade: Setting container_image for all crash 2024-02-27T06:28:10.046 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:10 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-27T06:28:10.046 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:10 smithi131 ceph-mon[66259]: Upgrade: osd.0 is safe to restart 2024-02-27T06:28:10.046 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:10 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/1295354089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:10.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:10 smithi150 ceph-mon[52803]: pgmap v54: 97 pgs: 97 active+clean; 336 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.9 MiB/s wr, 567 op/s 2024-02-27T06:28:10.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:10 smithi150 ceph-mon[52803]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:10.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:10 smithi150 ceph-mon[52803]: Upgrade: Setting container_image for all crash 2024-02-27T06:28:10.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:10 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-27T06:28:10.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:10 smithi150 ceph-mon[52803]: Upgrade: osd.0 is safe to restart 2024-02-27T06:28:10.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:10 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/1295354089' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:10.530 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:28:10.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 196581 11.37 MB/sec execute 288 sec latency 66.126 ms 2024-02-27T06:28:11.057 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:11 smithi131 ceph-mon[66259]: from='client.44151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:11.057 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:11 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/4159920248' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:28:11.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:11 smithi150 ceph-mon[52803]: from='client.44151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:11.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:11 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/4159920248' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:28:11.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 196951 11.37 MB/sec execute 289 sec latency 60.317 ms 2024-02-27T06:28:12.071 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:12 smithi131 ceph-mon[66259]: pgmap v55: 97 pgs: 97 active+clean; 336 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 408 op/s 2024-02-27T06:28:12.071 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:12 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:28:12.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:12 smithi150 ceph-mon[52803]: pgmap v55: 97 pgs: 97 active+clean; 336 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 408 op/s 2024-02-27T06:28:12.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:12 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:28:12.533 INFO:teuthology.orchestra.run.smithi131.stdout: 5 197491 11.38 MB/sec execute 290 sec latency 60.710 ms 2024-02-27T06:28:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: 5 197879 11.39 MB/sec execute 291 sec latency 31.640 ms 2024-02-27T06:28:13.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:13 smithi131 ceph-mon[66259]: pgmap v56: 97 pgs: 97 active+clean; 336 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 408 op/s 2024-02-27T06:28:13.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:13 smithi131 ceph-mon[66259]: Upgrade: Updating osd.0 2024-02-27T06:28:13.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:13 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:13.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:13 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T06:28:13.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:13 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:28:13.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:13 smithi131 ceph-mon[66259]: Deploying daemon osd.0 on smithi131 2024-02-27T06:28:13.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:13 smithi150 ceph-mon[52803]: pgmap v56: 97 pgs: 97 active+clean; 336 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 408 op/s 2024-02-27T06:28:13.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:13 smithi150 ceph-mon[52803]: Upgrade: Updating osd.0 2024-02-27T06:28:13.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:13 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:13.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:13 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-27T06:28:13.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:13 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:28:13.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:13 smithi150 ceph-mon[52803]: Deploying daemon osd.0 on smithi131 2024-02-27T06:28:14.534 INFO:teuthology.orchestra.run.smithi131.stdout: 5 198225 11.37 MB/sec execute 292 sec latency 47.561 ms 2024-02-27T06:28:15.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 198618 11.38 MB/sec execute 293 sec latency 37.138 ms 2024-02-27T06:28:15.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:15 smithi131 ceph-mon[66259]: pgmap v57: 97 pgs: 97 active+clean; 333 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.3 MiB/s wr, 646 op/s 2024-02-27T06:28:15.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:15 smithi150 ceph-mon[52803]: pgmap v57: 97 pgs: 97 active+clean; 333 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.3 MiB/s wr, 646 op/s 2024-02-27T06:28:16.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 198920 11.38 MB/sec execute 294 sec latency 59.114 ms 2024-02-27T06:28:17.476 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:17 smithi131 ceph-mon[66259]: pgmap v58: 97 pgs: 97 active+clean; 333 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.4 MiB/s wr, 436 op/s 2024-02-27T06:28:17.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 199173 11.36 MB/sec execute 295 sec latency 72.828 ms 2024-02-27T06:28:17.731 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:17 smithi131 systemd[1]: Stopping Ceph osd.0 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:28:17.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:17 smithi150 ceph-mon[52803]: pgmap v58: 97 pgs: 97 active+clean; 333 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.4 MiB/s wr, 436 op/s 2024-02-27T06:28:17.990 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:17 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0[35235]: 2024-02-27T06:28:17.933+0000 7fbd20b1d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:28:17.991 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:17 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0[35235]: 2024-02-27T06:28:17.933+0000 7fbd20b1d700 -1 osd.0 73 *** Got signal Terminated *** 2024-02-27T06:28:17.991 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:17 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0[35235]: 2024-02-27T06:28:17.933+0000 7fbd20b1d700 -1 osd.0 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:28:18.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 199473 11.36 MB/sec execute 296 sec latency 86.274 ms 2024-02-27T06:28:18.603 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:18 smithi131 ceph-mon[66259]: osd.0 marked itself down and dead 2024-02-27T06:28:18.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:18 smithi150 ceph-mon[52803]: osd.0 marked itself down and dead 2024-02-27T06:28:19.448 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:19 smithi131 podman[71938]: 2024-02-27 06:28:19.1035938 +0000 UTC m=+1.298413333 container died 09befbc8743b62e8fb869492827a28a481ebadc6048d2a368a9bb51e74d73dc6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_CLEAN=True, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, ceph=True) 2024-02-27T06:28:19.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 199483 11.32 MB/sec execute 297 sec latency 1051.974 ms 2024-02-27T06:28:19.629 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:19 smithi131 ceph-mon[66259]: pgmap v59: 97 pgs: 97 active+clean; 337 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.8 MiB/s wr, 651 op/s 2024-02-27T06:28:19.629 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:19 smithi131 ceph-mon[66259]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:28:19.629 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:19 smithi131 ceph-mon[66259]: osdmap e74: 8 total, 7 up, 8 in 2024-02-27T06:28:19.629 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:19 smithi131 podman[71938]: 2024-02-27 06:28:19.46460223 +0000 UTC m=+1.659421764 container cleanup 09befbc8743b62e8fb869492827a28a481ebadc6048d2a368a9bb51e74d73dc6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., RELEASE=HEAD, architecture=x86_64, name=centos-stream, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754) 2024-02-27T06:28:19.629 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:19 smithi131 bash[71938]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0 2024-02-27T06:28:19.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:19 smithi150 ceph-mon[52803]: pgmap v59: 97 pgs: 97 active+clean; 337 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.8 MiB/s wr, 651 op/s 2024-02-27T06:28:19.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:19 smithi150 ceph-mon[52803]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:28:19.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:19 smithi150 ceph-mon[52803]: osdmap e74: 8 total, 7 up, 8 in 2024-02-27T06:28:19.957 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:19 smithi131 podman[71957]: 2024-02-27 06:28:19.730691589 +0000 UTC m=+0.618860341 container remove 09befbc8743b62e8fb869492827a28a481ebadc6048d2a368a9bb51e74d73dc6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_BRANCH=HEAD, architecture=x86_64, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, vendor=Red Hat, Inc., version=8) 2024-02-27T06:28:20.461 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:20 smithi131 podman[72078]: 2024-02-27 06:28:20.175270199 +0000 UTC m=+0.019194162 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:28:20.461 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:20 smithi131 podman[72078]: 2024-02-27 06:28:20.335550399 +0000 UTC m=+0.179474345 container create 2356850f6b368c95a8232706c59e44250193a60e8d9877ea9acf8d0a499084aa (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-27T06:28:20.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 199803 11.29 MB/sec execute 298 sec latency 53.197 ms 2024-02-27T06:28:20.711 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:20 smithi131 ceph-mon[66259]: osdmap e75: 8 total, 7 up, 8 in 2024-02-27T06:28:20.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:20 smithi150 ceph-mon[52803]: osdmap e75: 8 total, 7 up, 8 in 2024-02-27T06:28:20.964 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:20 smithi131 podman[72078]: 2024-02-27 06:28:20.713334982 +0000 UTC m=+0.557258933 container init 2356850f6b368c95a8232706c59e44250193a60e8d9877ea9acf8d0a499084aa (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:28:20.964 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:20 smithi131 podman[72078]: 2024-02-27 06:28:20.724070021 +0000 UTC m=+0.567993977 container start 2356850f6b368c95a8232706c59e44250193a60e8d9877ea9acf8d0a499084aa (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:28:20.965 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:20 smithi131 podman[72078]: 2024-02-27 06:28:20.923631701 +0000 UTC m=+0.767555653 container attach 2356850f6b368c95a8232706c59e44250193a60e8d9877ea9acf8d0a499084aa (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:28:21.227 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:21 smithi131 podman[72078]: 2024-02-27 06:28:21.117138906 +0000 UTC m=+0.961062868 container died 2356850f6b368c95a8232706c59e44250193a60e8d9877ea9acf8d0a499084aa (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-02-27T06:28:21.486 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:21 smithi131 ceph-mon[66259]: pgmap v62: 97 pgs: 13 stale+active+clean, 84 active+clean; 337 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 679 op/s 2024-02-27T06:28:21.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 200068 11.27 MB/sec execute 299 sec latency 42.298 ms 2024-02-27T06:28:21.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:21 smithi150 ceph-mon[52803]: pgmap v62: 97 pgs: 13 stale+active+clean, 84 active+clean; 337 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 679 op/s 2024-02-27T06:28:22.251 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:22 smithi131 podman[72104]: 2024-02-27 06:28:22.074885845 +0000 UTC m=+0.950098555 container remove 2356850f6b368c95a8232706c59e44250193a60e8d9877ea9acf8d0a499084aa (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-27T06:28:22.252 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:22 smithi131 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-27T06:28:22.252 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:22 smithi131 systemd[1]: Stopped Ceph osd.0 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:28:22.252 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:22 smithi131 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.0.service: Consumed 3min 37.685s CPU time. 2024-02-27T06:28:22.511 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:22 smithi131 systemd[1]: Starting Ceph osd.0 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:28:22.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 200370 11.26 MB/sec execute 300 sec latency 45.561 ms 2024-02-27T06:28:22.763 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:22 smithi131 podman[72202]: 2024-02-27 06:28:22.592663953 +0000 UTC m=+0.025288750 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:28:22.763 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:22 smithi131 podman[72202]: 2024-02-27 06:28:22.722073242 +0000 UTC m=+0.154698022 container create 8117e08d7e9426b4ebe6fdf20827ad56ee56c905df14ba52baa67c3d75c5bf0c (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:28:23.275 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 podman[72202]: 2024-02-27 06:28:23.151726445 +0000 UTC m=+0.584351231 container init 8117e08d7e9426b4ebe6fdf20827ad56ee56c905df14ba52baa67c3d75c5bf0c (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS) 2024-02-27T06:28:23.275 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 podman[72202]: 2024-02-27 06:28:23.156112195 +0000 UTC m=+0.588736973 container start 8117e08d7e9426b4ebe6fdf20827ad56ee56c905df14ba52baa67c3d75c5bf0c (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:28:23.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 200779 11.26 MB/sec execute 301 sec latency 46.085 ms 2024-02-27T06:28:23.562 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 podman[72202]: 2024-02-27 06:28:23.330166291 +0000 UTC m=+0.762791067 container attach 8117e08d7e9426b4ebe6fdf20827ad56ee56c905df14ba52baa67c3d75c5bf0c (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-02-27T06:28:23.563 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:23 smithi131 ceph-mon[66259]: pgmap v63: 97 pgs: 13 stale+active+clean, 84 active+clean; 337 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.6 MiB/s wr, 322 op/s 2024-02-27T06:28:23.822 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate[72216]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate[72216]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 bash[72202]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 bash[72202]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 bash[72202]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 bash[72202]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 bash[72202]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 bash[72202]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 bash[72202]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate[72216]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate[72216]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate[72216]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate[72216]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate[72216]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-27T06:28:23.823 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:23 smithi131 podman[72348]: 2024-02-27 06:28:23.79634902 +0000 UTC m=+0.021307897 container died 8117e08d7e9426b4ebe6fdf20827ad56ee56c905df14ba52baa67c3d75c5bf0c (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-02-27T06:28:23.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:23 smithi150 ceph-mon[52803]: pgmap v63: 97 pgs: 13 stale+active+clean, 84 active+clean; 337 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.6 MiB/s wr, 322 op/s 2024-02-27T06:28:24.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 201122 11.25 MB/sec execute 302 sec latency 65.272 ms 2024-02-27T06:28:24.587 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:24 smithi131 ceph-mon[66259]: Health check failed: Degraded data redundancy: 204/1566 objects degraded (13.027%), 29 pgs degraded (PG_DEGRADED) 2024-02-27T06:28:24.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:24 smithi150 ceph-mon[52803]: Health check failed: Degraded data redundancy: 204/1566 objects degraded (13.027%), 29 pgs degraded (PG_DEGRADED) 2024-02-27T06:28:25.096 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:24 smithi131 podman[72348]: 2024-02-27 06:28:24.848641623 +0000 UTC m=+1.073600499 container remove 8117e08d7e9426b4ebe6fdf20827ad56ee56c905df14ba52baa67c3d75c5bf0c (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-27T06:28:25.096 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:25 smithi131 podman[72406]: 2024-02-27 06:28:24.98947652 +0000 UTC m=+0.020552705 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:28:25.373 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:25 smithi131 podman[72406]: 2024-02-27 06:28:25.123919777 +0000 UTC m=+0.154995959 container create 73ab15ca6f02c3d349c707d9736d218fdd0739f0820627cce57800c3d10ac44b (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:28:25.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 201408 11.25 MB/sec execute 303 sec latency 45.521 ms 2024-02-27T06:28:25.633 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:25 smithi131 ceph-mon[66259]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 329 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.5 MiB/s wr, 550 op/s; 204/1566 objects degraded (13.027%) 2024-02-27T06:28:25.633 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:25 smithi131 podman[72406]: 2024-02-27 06:28:25.472052744 +0000 UTC m=+0.503128933 container init 73ab15ca6f02c3d349c707d9736d218fdd0739f0820627cce57800c3d10ac44b (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:28:25.633 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:25 smithi131 podman[72406]: 2024-02-27 06:28:25.477442826 +0000 UTC m=+0.508519008 container start 73ab15ca6f02c3d349c707d9736d218fdd0739f0820627cce57800c3d10ac44b (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:28:25.633 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:25 smithi131 ceph-osd[72424]: -- 172.21.15.131:0/1470502983 <== mon.0 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556182cd9680 con 0x556182ca3000 2024-02-27T06:28:25.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:25 smithi150 ceph-mon[52803]: pgmap v64: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 329 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.5 MiB/s wr, 550 op/s; 204/1566 objects degraded (13.027%) 2024-02-27T06:28:26.149 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:25 smithi131 bash[72406]: 73ab15ca6f02c3d349c707d9736d218fdd0739f0820627cce57800c3d10ac44b 2024-02-27T06:28:26.149 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:25 smithi131 systemd[1]: Started Ceph osd.0 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:28:26.149 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:26 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0[72420]: 2024-02-27T06:28:26.035+0000 7fae1b2697c0 -1 Falling back to public interface 2024-02-27T06:28:26.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 201659 11.24 MB/sec execute 304 sec latency 79.335 ms 2024-02-27T06:28:27.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:27.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:27.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:28:27.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:27.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:28:27.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:27.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:27.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:28:27.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:27.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:28:27.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 202029 11.23 MB/sec execute 305 sec latency 50.845 ms 2024-02-27T06:28:28.180 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:27 smithi131 ceph-mon[66259]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 329 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.9 MiB/s wr, 227 op/s; 204/1566 objects degraded (13.027%) 2024-02-27T06:28:28.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:27 smithi150 ceph-mon[52803]: pgmap v65: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 329 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.9 MiB/s wr, 227 op/s; 204/1566 objects degraded (13.027%) 2024-02-27T06:28:28.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 202512 11.24 MB/sec execute 306 sec latency 35.206 ms 2024-02-27T06:28:29.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 202897 11.24 MB/sec execute 307 sec latency 32.486 ms 2024-02-27T06:28:30.200 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:29 smithi131 ceph-mon[66259]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 335 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.0 MiB/s wr, 411 op/s; 213/1644 objects degraded (12.956%) 2024-02-27T06:28:30.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:29 smithi150 ceph-mon[52803]: pgmap v66: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 335 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.0 MiB/s wr, 411 op/s; 213/1644 objects degraded (12.956%) 2024-02-27T06:28:30.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 203469 11.24 MB/sec execute 308 sec latency 46.620 ms 2024-02-27T06:28:31.206 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:30 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 213/1644 objects degraded (12.956%), 29 pgs degraded (PG_DEGRADED) 2024-02-27T06:28:31.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:30 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 213/1644 objects degraded (12.956%), 29 pgs degraded (PG_DEGRADED) 2024-02-27T06:28:31.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 203931 11.23 MB/sec execute 309 sec latency 82.526 ms 2024-02-27T06:28:32.012 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:31 smithi131 ceph-mon[66259]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 335 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.6 MiB/s wr, 373 op/s; 213/1644 objects degraded (12.956%) 2024-02-27T06:28:32.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:31 smithi150 ceph-mon[52803]: pgmap v67: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 335 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.6 MiB/s wr, 373 op/s; 213/1644 objects degraded (12.956%) 2024-02-27T06:28:32.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 204410 11.24 MB/sec execute 310 sec latency 51.350 ms 2024-02-27T06:28:33.535 INFO:teuthology.orchestra.run.smithi131.stdout: 5 204769 11.24 MB/sec execute 311 sec latency 64.418 ms 2024-02-27T06:28:33.764 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:33 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0[72420]: 2024-02-27T06:28:33.605+0000 7fae1b2697c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-02-27T06:28:34.015 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:33 smithi131 ceph-mon[66259]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 335 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.3 MiB/s wr, 341 op/s; 213/1644 objects degraded (12.956%) 2024-02-27T06:28:34.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:33 smithi150 ceph-mon[52803]: pgmap v68: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 335 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.3 MiB/s wr, 341 op/s; 213/1644 objects degraded (12.956%) 2024-02-27T06:28:34.536 INFO:teuthology.orchestra.run.smithi131.stdout: 5 205094 11.25 MB/sec execute 312 sec latency 70.127 ms 2024-02-27T06:28:35.275 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:35 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0[72420]: 2024-02-27T06:28:35.050+0000 7fae1b2697c0 -1 osd.0 73 log_to_monitors true 2024-02-27T06:28:35.541 INFO:teuthology.orchestra.run.smithi131.stdout: 5 205456 11.24 MB/sec execute 313 sec latency 44.380 ms 2024-02-27T06:28:36.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:35 smithi131 ceph-mon[66259]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 334 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.0 MiB/s wr, 636 op/s; 218/1713 objects degraded (12.726%) 2024-02-27T06:28:36.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:35 smithi131 ceph-mon[66259]: from='osd.0 [v2:172.21.15.131:6802/2081416784,v1:172.21.15.131:6803/2081416784]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T06:28:36.033 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:35 smithi131 ceph-mon[66259]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T06:28:36.034 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:35 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 218/1713 objects degraded (12.726%), 29 pgs degraded (PG_DEGRADED) 2024-02-27T06:28:36.034 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:35 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:36.034 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:35 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:35 smithi150 ceph-mon[52803]: pgmap v69: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 334 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 6.0 MiB/s wr, 636 op/s; 218/1713 objects degraded (12.726%) 2024-02-27T06:28:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:35 smithi150 ceph-mon[52803]: from='osd.0 [v2:172.21.15.131:6802/2081416784,v1:172.21.15.131:6803/2081416784]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T06:28:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:35 smithi150 ceph-mon[52803]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-27T06:28:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:35 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 218/1713 objects degraded (12.726%), 29 pgs degraded (PG_DEGRADED) 2024-02-27T06:28:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:35 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:35 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:36.536 INFO:teuthology.orchestra.run.smithi131.stdout: 5 205788 11.23 MB/sec execute 314 sec latency 140.154 ms 2024-02-27T06:28:36.560 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:28:36 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0[72420]: 2024-02-27T06:28:36.538+0000 7fae086b9700 -1 osd.0 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:28:37.063 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:37 smithi131 ceph-mon[66259]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-27T06:28:37.063 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:37 smithi131 ceph-mon[66259]: osdmap e76: 8 total, 7 up, 8 in 2024-02-27T06:28:37.063 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:37 smithi131 ceph-mon[66259]: from='osd.0 [v2:172.21.15.131:6802/2081416784,v1:172.21.15.131:6803/2081416784]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:28:37.064 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:37 smithi131 ceph-mon[66259]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:28:37.064 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:37 smithi131 ceph-mon[66259]: from='osd.0 ' entity='osd.0' 2024-02-27T06:28:37.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:37 smithi150 ceph-mon[52803]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-27T06:28:37.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:37 smithi150 ceph-mon[52803]: osdmap e76: 8 total, 7 up, 8 in 2024-02-27T06:28:37.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:37 smithi150 ceph-mon[52803]: from='osd.0 [v2:172.21.15.131:6802/2081416784,v1:172.21.15.131:6803/2081416784]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:28:37.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:37 smithi150 ceph-mon[52803]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:28:37.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:37 smithi150 ceph-mon[52803]: from='osd.0 ' entity='osd.0' 2024-02-27T06:28:37.536 INFO:teuthology.orchestra.run.smithi131.stdout: 5 206124 11.24 MB/sec execute 315 sec latency 35.478 ms 2024-02-27T06:28:38.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:38 smithi131 ceph-mon[66259]: pgmap v71: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 334 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.7 MiB/s wr, 581 op/s; 218/1713 objects degraded (12.726%) 2024-02-27T06:28:38.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:38 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:38.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:38 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:38.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:38 smithi131 ceph-mon[66259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:28:38.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:38 smithi131 ceph-mon[66259]: osd.0 [v2:172.21.15.131:6802/2081416784,v1:172.21.15.131:6803/2081416784] boot 2024-02-27T06:28:38.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:38 smithi131 ceph-mon[66259]: osdmap e77: 8 total, 8 up, 8 in 2024-02-27T06:28:38.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:38 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:28:38.536 INFO:teuthology.orchestra.run.smithi131.stdout: 5 206205 11.20 MB/sec execute 316 sec latency 1018.012 ms 2024-02-27T06:28:38.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:38 smithi150 ceph-mon[52803]: pgmap v71: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 334 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.7 MiB/s wr, 581 op/s; 218/1713 objects degraded (12.726%) 2024-02-27T06:28:38.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:38 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:38.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:38 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:38.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:38 smithi150 ceph-mon[52803]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:28:38.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:38 smithi150 ceph-mon[52803]: osd.0 [v2:172.21.15.131:6802/2081416784,v1:172.21.15.131:6803/2081416784] boot 2024-02-27T06:28:38.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:38 smithi150 ceph-mon[52803]: osdmap e77: 8 total, 8 up, 8 in 2024-02-27T06:28:38.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:38 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-27T06:28:39.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 206216 11.17 MB/sec execute 317 sec latency 2021.732 ms 2024-02-27T06:28:39.699 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:39 smithi131 ceph-mon[66259]: pgmap v73: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 760 op/s; 211/1713 objects degraded (12.318%) 2024-02-27T06:28:39.699 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:39 smithi131 ceph-mon[66259]: osdmap e78: 8 total, 8 up, 8 in 2024-02-27T06:28:39.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:39 smithi150 ceph-mon[52803]: pgmap v73: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 760 op/s; 211/1713 objects degraded (12.318%) 2024-02-27T06:28:39.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:39 smithi150 ceph-mon[52803]: osdmap e78: 8 total, 8 up, 8 in 2024-02-27T06:28:40.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 206544 11.15 MB/sec execute 318 sec latency 2089.642 ms 2024-02-27T06:28:40.725 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:40 smithi131 ceph-mon[66259]: osdmap e79: 8 total, 8 up, 8 in 2024-02-27T06:28:40.725 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:40 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 211/1713 objects degraded (12.318%), 29 pgs degraded (PG_DEGRADED) 2024-02-27T06:28:40.725 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:40 smithi131 ceph-mon[66259]: osdmap e80: 8 total, 8 up, 8 in 2024-02-27T06:28:40.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:40 smithi150 ceph-mon[52803]: osdmap e79: 8 total, 8 up, 8 in 2024-02-27T06:28:40.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:40 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 211/1713 objects degraded (12.318%), 29 pgs degraded (PG_DEGRADED) 2024-02-27T06:28:40.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:40 smithi150 ceph-mon[52803]: osdmap e80: 8 total, 8 up, 8 in 2024-02-27T06:28:41.070 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:28:41.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 206945 11.15 MB/sec execute 319 sec latency 49.113 ms 2024-02-27T06:28:41.742 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:41 smithi131 ceph-mon[66259]: pgmap v76: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.6 MiB/s wr, 576 op/s; 211/1713 objects degraded (12.318%) 2024-02-27T06:28:41.743 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:41 smithi131 ceph-mon[66259]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:41.743 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:41 smithi131 ceph-mon[66259]: osdmap e81: 8 total, 8 up, 8 in 2024-02-27T06:28:41.743 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:41.743 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:28:41.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:41 smithi150 ceph-mon[52803]: pgmap v76: 97 pgs: 29 active+undersized+degraded, 7 active+undersized, 61 active+clean; 341 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 6.6 MiB/s wr, 576 op/s; 211/1713 objects degraded (12.318%) 2024-02-27T06:28:41.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:41 smithi150 ceph-mon[52803]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:41.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:41 smithi150 ceph-mon[52803]: osdmap e81: 8 total, 8 up, 8 in 2024-02-27T06:28:41.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:41.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:28:42.077 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:28:42.077 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (4m) 6s ago 13m 22.6M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (57s) 6s ago 13m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (50s) 45s ago 13m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (4m) 6s ago 13m 53.1M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (7m) 6s ago 7m 38.5M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (7m) 45s ago 7m 14.2M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (3m) 6s ago 14m 499M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (2m) 45s ago 13m 449M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (2m) 6s ago 14m 45.3M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (116s) 45s ago 13m 36.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (7m) 6s ago 7m 268M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (7m) 45s ago 7m 81.4M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (4m) 6s ago 13m 17.8M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (3m) 45s ago 12m 17.2M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (17s) 6s ago 12m 12.0M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:28:42.078 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (11m) 6s ago 11m 354M 1936M 17.2.0 e1d6a67b021e 50b42cff9be9 2024-02-27T06:28:42.079 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (11m) 6s ago 11m 497M 1936M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:28:42.079 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (10m) 6s ago 10m 346M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:28:42.079 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (10m) 45s ago 10m 557M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:28:42.079 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (10m) 45s ago 10m 436M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:28:42.079 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (9m) 45s ago 9m 347M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:28:42.079 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (9m) 45s ago 9m 542M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:28:42.079 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (2m) 6s ago 12m 65.7M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:28:42.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 207213 11.13 MB/sec execute 320 sec latency 870.408 ms 2024-02-27T06:28:42.665 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:28:42.666 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-02-27T06:28:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 5 2024-02-27T06:28:42.667 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:28:42.667 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:28:42.764 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:42.764 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='client.34234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:42.764 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:42.764 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:42.764 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:28:42.764 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:28:42.764 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:42.764 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:42.764 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: osdmap e82: 8 total, 8 up, 8 in 2024-02-27T06:28:42.764 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:28:42.764 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:42.765 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:42.765 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:42.765 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T06:28:42.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='client.34230 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:42.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='client.34234 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:42.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:42.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:42.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:28:42.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:28:42.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:42.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:42.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: osdmap e82: 8 total, 8 up, 8 in 2024-02-27T06:28:42.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:28:42.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:42.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:42.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:42.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T06:28:43.161 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:28:43.161 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:28:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:28:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:28:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:28:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:28:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:28:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:28:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:28:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "7/23 daemons upgraded", 2024-02-27T06:28:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T06:28:43.162 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:28:43.162 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:28:43.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 207525 11.11 MB/sec execute 321 sec latency 978.215 ms 2024-02-27T06:28:43.734 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_WARN Degraded data redundancy: 191/1719 objects degraded (11.111%), 26 pgs degraded 2024-02-27T06:28:43.734 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 191/1719 objects degraded (11.111%), 26 pgs degraded 2024-02-27T06:28:43.734 INFO:teuthology.orchestra.run.smithi131.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-02-27T06:28:43.734 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-02-27T06:28:43.734 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.1 is active+undersized+degraded, acting [2,3] 2024-02-27T06:28:43.734 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.4 is active+undersized+degraded, acting [1,7] 2024-02-27T06:28:43.734 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.5 is active+undersized+degraded, acting [7,4] 2024-02-27T06:28:43.734 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.8 is active+undersized+degraded, acting [7,1] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.c is active+undersized+degraded, acting [2,5] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.f is active+undersized+degraded, acting [4,2] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.15 is active+undersized+degraded, acting [1,5] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.19 is active+undersized+degraded, acting [3,6] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.1d is active+undersized+degraded, acting [7,6] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.1e is active+undersized+degraded, acting [3,5] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2024-02-27T06:28:43.735 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-02-27T06:28:43.736 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-02-27T06:28:43.736 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-02-27T06:28:43.736 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-02-27T06:28:43.736 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-02-27T06:28:43.736 INFO:teuthology.orchestra.run.smithi131.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-02-27T06:28:43.736 INFO:teuthology.orchestra.run.smithi131.stdout: pg 4.19 is active+undersized+degraded, acting [3,2] 2024-02-27T06:28:43.772 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:43 smithi131 ceph-mon[66259]: pgmap v79: 97 pgs: 26 active+undersized+degraded, 2 active+undersized, 69 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 48 KiB/s wr, 2 op/s; 191/1719 objects degraded (11.111%); 0 B/s, 3 objects/s recovering 2024-02-27T06:28:43.772 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:43 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T06:28:43.772 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:43 smithi131 ceph-mon[66259]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-02-27T06:28:43.772 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:43 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/2465003370' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:43.772 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:43 smithi131 ceph-mon[66259]: from='client.34242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:43.772 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:43 smithi131 ceph-mon[66259]: osdmap e83: 8 total, 8 up, 8 in 2024-02-27T06:28:43.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:43 smithi150 ceph-mon[52803]: pgmap v79: 97 pgs: 26 active+undersized+degraded, 2 active+undersized, 69 active+clean; 342 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 48 KiB/s wr, 2 op/s; 191/1719 objects degraded (11.111%); 0 B/s, 3 objects/s recovering 2024-02-27T06:28:43.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:43 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T06:28:43.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:43 smithi150 ceph-mon[52803]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-02-27T06:28:43.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:43 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/2465003370' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:28:43.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:43 smithi150 ceph-mon[52803]: from='client.34242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:28:43.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:43 smithi150 ceph-mon[52803]: osdmap e83: 8 total, 8 up, 8 in 2024-02-27T06:28:44.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 207734 11.10 MB/sec execute 322 sec latency 881.512 ms 2024-02-27T06:28:44.788 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:44 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/4231978794' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:28:44.788 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:44 smithi131 ceph-mon[66259]: osdmap e84: 8 total, 8 up, 8 in 2024-02-27T06:28:44.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:44 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/4231978794' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:28:44.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:44 smithi150 ceph-mon[52803]: osdmap e84: 8 total, 8 up, 8 in 2024-02-27T06:28:45.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 208160 11.11 MB/sec execute 323 sec latency 846.318 ms 2024-02-27T06:28:45.814 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:45 smithi131 ceph-mon[66259]: pgmap v82: 97 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.1 MiB/s wr, 414 op/s; 159/1668 objects degraded (9.532%); 1.1 MiB/s, 20 objects/s recovering 2024-02-27T06:28:45.814 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:45 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 159/1668 objects degraded (9.532%), 9 pgs degraded (PG_DEGRADED) 2024-02-27T06:28:45.814 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:45 smithi131 ceph-mon[66259]: osdmap e85: 8 total, 8 up, 8 in 2024-02-27T06:28:45.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:45 smithi150 ceph-mon[52803]: pgmap v82: 97 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.1 MiB/s wr, 414 op/s; 159/1668 objects degraded (9.532%); 1.1 MiB/s, 20 objects/s recovering 2024-02-27T06:28:45.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:45 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 159/1668 objects degraded (9.532%), 9 pgs degraded (PG_DEGRADED) 2024-02-27T06:28:45.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:45 smithi150 ceph-mon[52803]: osdmap e85: 8 total, 8 up, 8 in 2024-02-27T06:28:46.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 208559 11.11 MB/sec execute 324 sec latency 40.587 ms 2024-02-27T06:28:47.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 208872 11.12 MB/sec execute 325 sec latency 74.879 ms 2024-02-27T06:28:47.632 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:47 smithi131 ceph-mon[66259]: pgmap v85: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 92 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 471 op/s; 71/1650 objects degraded (4.303%); 1.1 MiB/s, 45 objects/s recovering 2024-02-27T06:28:47.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:47 smithi150 ceph-mon[52803]: pgmap v85: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 92 active+clean; 332 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.3 MiB/s wr, 471 op/s; 71/1650 objects degraded (4.303%); 1.1 MiB/s, 45 objects/s recovering 2024-02-27T06:28:48.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 209121 11.10 MB/sec execute 326 sec latency 61.182 ms 2024-02-27T06:28:48.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:48 smithi131 ceph-mon[66259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/1650 objects degraded (4.303%), 4 pgs degraded) 2024-02-27T06:28:48.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:48 smithi131 ceph-mon[66259]: Cluster is now healthy 2024-02-27T06:28:49.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:48 smithi150 ceph-mon[52803]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/1650 objects degraded (4.303%), 4 pgs degraded) 2024-02-27T06:28:49.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:48 smithi150 ceph-mon[52803]: Cluster is now healthy 2024-02-27T06:28:49.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 209353 11.08 MB/sec execute 327 sec latency 40.641 ms 2024-02-27T06:28:49.685 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:49 smithi131 ceph-mon[66259]: pgmap v86: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.6 MiB/s wr, 605 op/s; 726 KiB/s, 36 objects/s recovering 2024-02-27T06:28:50.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:49 smithi150 ceph-mon[52803]: pgmap v86: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.6 MiB/s wr, 605 op/s; 726 KiB/s, 36 objects/s recovering 2024-02-27T06:28:50.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 209566 11.06 MB/sec execute 328 sec latency 73.187 ms 2024-02-27T06:28:51.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 209815 11.05 MB/sec execute 329 sec latency 47.067 ms 2024-02-27T06:28:51.789 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:51 smithi131 ceph-mon[66259]: pgmap v87: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.6 MiB/s wr, 343 op/s; 0 B/s, 27 objects/s recovering 2024-02-27T06:28:52.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:51 smithi150 ceph-mon[52803]: pgmap v87: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.6 MiB/s wr, 343 op/s; 0 B/s, 27 objects/s recovering 2024-02-27T06:28:52.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 210080 11.04 MB/sec execute 330 sec latency 52.729 ms 2024-02-27T06:28:53.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 210427 11.04 MB/sec execute 331 sec latency 65.683 ms 2024-02-27T06:28:53.829 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:53 smithi131 ceph-mon[66259]: pgmap v88: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.5 MiB/s wr, 311 op/s; 0 B/s, 23 objects/s recovering 2024-02-27T06:28:54.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:53 smithi150 ceph-mon[52803]: pgmap v88: 97 pgs: 97 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.5 MiB/s wr, 311 op/s; 0 B/s, 23 objects/s recovering 2024-02-27T06:28:54.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 210704 11.02 MB/sec execute 332 sec latency 93.677 ms 2024-02-27T06:28:55.541 INFO:teuthology.orchestra.run.smithi131.stdout: 5 211119 11.01 MB/sec execute 333 sec latency 35.519 ms 2024-02-27T06:28:55.863 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:55 smithi131 ceph-mon[66259]: pgmap v89: 97 pgs: 97 active+clean; 330 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 433 op/s; 0 B/s, 19 objects/s recovering 2024-02-27T06:28:56.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:55 smithi150 ceph-mon[52803]: pgmap v89: 97 pgs: 97 active+clean; 330 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 433 op/s; 0 B/s, 19 objects/s recovering 2024-02-27T06:28:56.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 211507 11.00 MB/sec execute 334 sec latency 40.460 ms 2024-02-27T06:28:57.540 INFO:teuthology.orchestra.run.smithi131.stdout: 5 211934 11.02 MB/sec execute 335 sec latency 51.076 ms 2024-02-27T06:28:57.648 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:57 smithi131 ceph-mon[66259]: pgmap v90: 97 pgs: 97 active+clean; 329 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.5 MiB/s wr, 413 op/s; 0 B/s, 17 objects/s recovering 2024-02-27T06:28:57.648 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:57.648 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:28:57.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:57.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:57 smithi150 ceph-mon[52803]: pgmap v90: 97 pgs: 97 active+clean; 329 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.5 MiB/s wr, 413 op/s; 0 B/s, 17 objects/s recovering 2024-02-27T06:28:57.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:57.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:28:57.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:28:58.541 INFO:teuthology.orchestra.run.smithi131.stdout: 5 212243 11.03 MB/sec execute 336 sec latency 56.024 ms 2024-02-27T06:28:58.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T06:28:58.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:58 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T06:28:58.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:58 smithi131 ceph-mon[66259]: Upgrade: osd.1 is safe to restart 2024-02-27T06:28:58.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T06:28:58.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:58 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-27T06:28:58.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:58 smithi150 ceph-mon[52803]: Upgrade: osd.1 is safe to restart 2024-02-27T06:28:59.541 INFO:teuthology.orchestra.run.smithi131.stdout: 5 212551 11.02 MB/sec execute 337 sec latency 45.654 ms 2024-02-27T06:28:59.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:28:59 smithi131 ceph-mon[66259]: pgmap v91: 97 pgs: 97 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.3 MiB/s wr, 557 op/s; 0 B/s, 6 objects/s recovering 2024-02-27T06:28:59.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:28:59 smithi150 ceph-mon[52803]: pgmap v91: 97 pgs: 97 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.3 MiB/s wr, 557 op/s; 0 B/s, 6 objects/s recovering 2024-02-27T06:29:00.541 INFO:teuthology.orchestra.run.smithi131.stdout: 5 212867 11.00 MB/sec execute 338 sec latency 87.768 ms 2024-02-27T06:29:01.541 INFO:teuthology.orchestra.run.smithi131.stdout: 5 213176 10.99 MB/sec execute 339 sec latency 37.504 ms 2024-02-27T06:29:01.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:01 smithi131 ceph-mon[66259]: pgmap v92: 97 pgs: 97 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 378 op/s 2024-02-27T06:29:01.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:01 smithi131 ceph-mon[66259]: Upgrade: Updating osd.1 2024-02-27T06:29:01.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:01.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T06:29:01.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:29:01.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:01 smithi131 ceph-mon[66259]: Deploying daemon osd.1 on smithi131 2024-02-27T06:29:02.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:01 smithi150 ceph-mon[52803]: pgmap v92: 97 pgs: 97 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.6 MiB/s wr, 378 op/s 2024-02-27T06:29:02.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:01 smithi150 ceph-mon[52803]: Upgrade: Updating osd.1 2024-02-27T06:29:02.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:02.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-27T06:29:02.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:29:02.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:01 smithi150 ceph-mon[52803]: Deploying daemon osd.1 on smithi131 2024-02-27T06:29:02.541 INFO:teuthology.orchestra.run.smithi131.stdout: 5 213582 10.99 MB/sec execute 340 sec latency 62.927 ms 2024-02-27T06:29:03.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 214043 11.00 MB/sec execute 341 sec latency 53.738 ms 2024-02-27T06:29:03.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:03 smithi131 ceph-mon[66259]: pgmap v93: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 393 op/s 2024-02-27T06:29:04.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:03 smithi150 ceph-mon[52803]: pgmap v93: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 393 op/s 2024-02-27T06:29:04.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 214473 10.98 MB/sec execute 342 sec latency 37.231 ms 2024-02-27T06:29:05.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 214847 10.98 MB/sec execute 343 sec latency 49.484 ms 2024-02-27T06:29:05.755 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:05 smithi131 ceph-mon[66259]: pgmap v94: 97 pgs: 97 active+clean; 334 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.7 MiB/s wr, 581 op/s 2024-02-27T06:29:06.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:05 smithi150 ceph-mon[52803]: pgmap v94: 97 pgs: 97 active+clean; 334 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.7 KiB/s rd, 5.7 MiB/s wr, 581 op/s 2024-02-27T06:29:06.258 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:06 smithi131 systemd[1]: Stopping Ceph osd.1 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:29:06.516 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:06 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[39275]: 2024-02-27T06:29:06.476+0000 7f9e47a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:29:06.517 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:06 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[39275]: 2024-02-27T06:29:06.476+0000 7f9e47a00700 -1 osd.1 85 *** Got signal Terminated *** 2024-02-27T06:29:06.517 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:06 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[39275]: 2024-02-27T06:29:06.476+0000 7f9e47a00700 -1 osd.1 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:29:06.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 215239 10.98 MB/sec execute 344 sec latency 36.506 ms 2024-02-27T06:29:06.767 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:06 smithi131 ceph-mon[66259]: osd.1 marked itself down and dead 2024-02-27T06:29:07.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:06 smithi150 ceph-mon[52803]: osd.1 marked itself down and dead 2024-02-27T06:29:07.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 215393 10.97 MB/sec execute 345 sec latency 733.320 ms 2024-02-27T06:29:07.676 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:07 smithi131 podman[74549]: 2024-02-27 06:29:07.381485286 +0000 UTC m=+1.014632345 container died 50b42cff9be99c71d5fc7b754021d2bb2c3336e71bffd680163ae621f15a8dcc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., vcs-type=git, GIT_BRANCH=HEAD, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-27T06:29:07.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:07 smithi131 ceph-mon[66259]: pgmap v95: 97 pgs: 97 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.6 MiB/s wr, 448 op/s 2024-02-27T06:29:07.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:07 smithi131 ceph-mon[66259]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:29:07.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:07 smithi131 ceph-mon[66259]: osdmap e86: 8 total, 7 up, 8 in 2024-02-27T06:29:07.927 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:07 smithi131 podman[74549]: 2024-02-27 06:29:07.859249809 +0000 UTC m=+1.492396869 container cleanup 50b42cff9be99c71d5fc7b754021d2bb2c3336e71bffd680163ae621f15a8dcc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, architecture=x86_64, io.openshift.expose-services=, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , ceph=True, io.k8s.display-name=CentOS Stream 8, version=8, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, distribution-scope=public) 2024-02-27T06:29:07.927 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:07 smithi131 bash[74549]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1 2024-02-27T06:29:08.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:07 smithi150 ceph-mon[52803]: pgmap v95: 97 pgs: 97 active+clean; 333 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.6 MiB/s wr, 448 op/s 2024-02-27T06:29:08.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:07 smithi150 ceph-mon[52803]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:29:08.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:07 smithi150 ceph-mon[52803]: osdmap e86: 8 total, 7 up, 8 in 2024-02-27T06:29:08.433 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:08 smithi131 podman[74565]: 2024-02-27 06:29:08.404470346 +0000 UTC m=+1.026406651 container remove 50b42cff9be99c71d5fc7b754021d2bb2c3336e71bffd680163ae621f15a8dcc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1, distribution-scope=public, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, vcs-type=git, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.openshift.expose-services=, vendor=Red Hat, Inc., version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-27T06:29:08.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 215725 10.97 MB/sec execute 346 sec latency 995.077 ms 2024-02-27T06:29:08.943 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:08 smithi131 ceph-mon[66259]: osdmap e87: 8 total, 7 up, 8 in 2024-02-27T06:29:08.944 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:08 smithi131 podman[74649]: 2024-02-27 06:29:08.780830208 +0000 UTC m=+0.028266640 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:29:08.944 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:08 smithi131 podman[74649]: 2024-02-27 06:29:08.89743189 +0000 UTC m=+0.144868305 container create a4b233380ad37608cdc3aeab4a73503e84697d5a5f2ace86d2d316ca53d8ee79 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-02-27T06:29:09.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:08 smithi150 ceph-mon[52803]: osdmap e87: 8 total, 7 up, 8 in 2024-02-27T06:29:09.455 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:09 smithi131 podman[74649]: 2024-02-27 06:29:09.271314508 +0000 UTC m=+0.518750924 container init a4b233380ad37608cdc3aeab4a73503e84697d5a5f2ace86d2d316ca53d8ee79 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:29:09.455 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:09 smithi131 podman[74649]: 2024-02-27 06:29:09.281401648 +0000 UTC m=+0.528838072 container start a4b233380ad37608cdc3aeab4a73503e84697d5a5f2ace86d2d316ca53d8ee79 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-02-27T06:29:09.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 216193 10.98 MB/sec execute 347 sec latency 36.482 ms 2024-02-27T06:29:09.706 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:09 smithi131 podman[74649]: 2024-02-27 06:29:09.452944862 +0000 UTC m=+0.700381283 container attach a4b233380ad37608cdc3aeab4a73503e84697d5a5f2ace86d2d316ca53d8ee79 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=) 2024-02-27T06:29:09.957 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:09 smithi131 podman[74649]: 2024-02-27 06:29:09.742916669 +0000 UTC m=+0.990353090 container died a4b233380ad37608cdc3aeab4a73503e84697d5a5f2ace86d2d316ca53d8ee79 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222) 2024-02-27T06:29:09.957 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:09 smithi131 ceph-mon[66259]: pgmap v98: 97 pgs: 35 peering, 1 stale+active+clean, 61 active+clean; 333 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 637 op/s 2024-02-27T06:29:09.957 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:09 smithi131 ceph-mon[66259]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-02-27T06:29:10.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:09 smithi150 ceph-mon[52803]: pgmap v98: 97 pgs: 35 peering, 1 stale+active+clean, 61 active+clean; 333 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 637 op/s 2024-02-27T06:29:10.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:09 smithi150 ceph-mon[52803]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-02-27T06:29:10.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 216644 10.97 MB/sec execute 348 sec latency 35.420 ms 2024-02-27T06:29:10.967 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:10 smithi131 podman[74675]: 2024-02-27 06:29:10.906931868 +0000 UTC m=+1.155336972 container remove a4b233380ad37608cdc3aeab4a73503e84697d5a5f2ace86d2d316ca53d8ee79 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-27T06:29:10.967 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:10 smithi131 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-27T06:29:10.967 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:10 smithi131 systemd[1]: Stopped Ceph osd.1 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:29:10.967 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:10 smithi131 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.1.service: Consumed 4min 6.969s CPU time. 2024-02-27T06:29:11.475 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:11 smithi131 systemd[1]: Starting Ceph osd.1 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:29:11.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 216998 10.97 MB/sec execute 349 sec latency 36.695 ms 2024-02-27T06:29:11.728 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:11 smithi131 podman[74768]: 2024-02-27 06:29:11.498018717 +0000 UTC m=+0.044217331 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:29:11.729 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:11 smithi131 podman[74768]: 2024-02-27 06:29:11.659173391 +0000 UTC m=+0.205371983 container create 5e3f508c35507c99ee6712348c6ce432689fd6b9d95b31cb66746ac4987e8188 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-27T06:29:11.983 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:11 smithi131 podman[74768]: 2024-02-27 06:29:11.954390312 +0000 UTC m=+0.500588912 container init 5e3f508c35507c99ee6712348c6ce432689fd6b9d95b31cb66746ac4987e8188 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:29:11.983 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:11 smithi131 podman[74768]: 2024-02-27 06:29:11.965292283 +0000 UTC m=+0.511490882 container start 5e3f508c35507c99ee6712348c6ce432689fd6b9d95b31cb66746ac4987e8188 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:29:11.983 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:11 smithi131 ceph-mon[66259]: pgmap v99: 97 pgs: 35 peering, 1 stale+active+clean, 61 active+clean; 333 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 5.8 MiB/s wr, 615 op/s 2024-02-27T06:29:11.983 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:11.983 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:29:12.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:11 smithi150 ceph-mon[52803]: pgmap v99: 97 pgs: 35 peering, 1 stale+active+clean, 61 active+clean; 333 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 5.8 MiB/s wr, 615 op/s 2024-02-27T06:29:12.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:12.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:29:12.233 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 podman[74768]: 2024-02-27 06:29:12.096832136 +0000 UTC m=+0.643030735 container attach 5e3f508c35507c99ee6712348c6ce432689fd6b9d95b31cb66746ac4987e8188 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-27T06:29:12.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 217336 10.96 MB/sec execute 350 sec latency 62.804 ms 2024-02-27T06:29:12.736 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate[74780]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-27T06:29:12.736 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate[74780]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-27T06:29:12.736 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 bash[74768]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-27T06:29:12.736 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 bash[74768]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-27T06:29:12.736 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 bash[74768]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-27T06:29:12.736 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 bash[74768]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-27T06:29:12.736 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 bash[74768]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-27T06:29:12.736 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 bash[74768]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-27T06:29:12.736 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 bash[74768]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-27T06:29:12.737 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate[74780]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-27T06:29:12.737 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate[74780]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-27T06:29:12.737 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate[74780]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-27T06:29:12.737 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate[74780]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-27T06:29:12.737 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate[74780]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-27T06:29:12.737 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:12 smithi131 podman[74768]: 2024-02-27 06:29:12.665645514 +0000 UTC m=+1.211844112 container died 5e3f508c35507c99ee6712348c6ce432689fd6b9d95b31cb66746ac4987e8188 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-27T06:29:12.986 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:12 smithi131 ceph-mon[66259]: Health check failed: Degraded data redundancy: 20/1647 objects degraded (1.214%), 2 pgs degraded (PG_DEGRADED) 2024-02-27T06:29:13.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:12 smithi150 ceph-mon[52803]: Health check failed: Degraded data redundancy: 20/1647 objects degraded (1.214%), 2 pgs degraded (PG_DEGRADED) 2024-02-27T06:29:13.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 217760 10.96 MB/sec execute 351 sec latency 42.044 ms 2024-02-27T06:29:13.768 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:13 smithi131 podman[74917]: 2024-02-27 06:29:13.686029995 +0000 UTC m=+1.010878621 container remove 5e3f508c35507c99ee6712348c6ce432689fd6b9d95b31cb66746ac4987e8188 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD) 2024-02-27T06:29:14.018 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:13 smithi131 podman[74978]: 2024-02-27 06:29:13.848860382 +0000 UTC m=+0.023323591 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:29:14.019 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:13 smithi131 ceph-mon[66259]: pgmap v100: 97 pgs: 2 active+undersized+degraded, 35 peering, 2 active+undersized, 58 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.0 MiB/s wr, 471 op/s; 20/1647 objects degraded (1.214%) 2024-02-27T06:29:14.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:13 smithi150 ceph-mon[52803]: pgmap v100: 97 pgs: 2 active+undersized+degraded, 35 peering, 2 active+undersized, 58 active+clean; 338 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.0 MiB/s wr, 471 op/s; 20/1647 objects degraded (1.214%) 2024-02-27T06:29:14.251 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:29:14.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 218125 10.95 MB/sec execute 352 sec latency 41.658 ms 2024-02-27T06:29:14.772 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:14 smithi131 podman[74978]: 2024-02-27 06:29:14.514356122 +0000 UTC m=+0.688819320 container create be4665f197cb4ae33a66fbbb1f04d744fc982c7f1e880d8360af92a1f0ec995f (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:29:15.026 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:14 smithi131 ceph-mon[66259]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:15.026 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:14 smithi131 podman[74978]: 2024-02-27 06:29:14.887762694 +0000 UTC m=+1.062225890 container init be4665f197cb4ae33a66fbbb1f04d744fc982c7f1e880d8360af92a1f0ec995f (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:29:15.027 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:14 smithi131 podman[74978]: 2024-02-27 06:29:14.894036187 +0000 UTC m=+1.068499387 container start be4665f197cb4ae33a66fbbb1f04d744fc982c7f1e880d8360af92a1f0ec995f (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-27T06:29:15.027 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:14 smithi131 ceph-osd[75063]: -- 172.21.15.131:0/3812086287 <== mon.0 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x565261208000 con 0x56526042cc00 2024-02-27T06:29:15.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:14 smithi150 ceph-mon[52803]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:15.245 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:29:15.245 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (5m) 39s ago 14m 22.6M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:29:15.245 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (90s) 39s ago 14m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:29:15.245 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (83s) 78s ago 13m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:29:15.245 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (4m) 39s ago 13m 53.1M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (8m) 39s ago 8m 38.5M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (8m) 78s ago 8m 14.2M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (4m) 39s ago 15m 499M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (3m) 78s ago 13m 449M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (3m) 39s ago 15m 45.3M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (2m) 78s ago 13m 36.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (8m) 39s ago 8m 268M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (8m) 78s ago 7m 81.4M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (4m) 39s ago 13m 17.8M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (4m) 78s ago 13m 17.2M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (50s) 39s ago 12m 12.0M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 starting - - - 1936M 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (11m) 39s ago 11m 497M 1936M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:29:15.246 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (11m) 39s ago 11m 346M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:29:15.247 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (11m) 78s ago 11m 557M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:29:15.247 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (10m) 78s ago 10m 436M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:29:15.247 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (10m) 78s ago 10m 347M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:29:15.247 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (9m) 78s ago 9m 542M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:29:15.247 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (3m) 39s ago 13m 65.7M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:29:15.280 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:15 smithi131 bash[74978]: be4665f197cb4ae33a66fbbb1f04d744fc982c7f1e880d8360af92a1f0ec995f 2024-02-27T06:29:15.280 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:15 smithi131 systemd[1]: Started Ceph osd.1 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:29:15.531 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:15 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[75036]: 2024-02-27T06:29:15.477+0000 7f375d75c7c0 -1 Falling back to public interface 2024-02-27T06:29:15.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 218430 10.94 MB/sec execute 353 sec latency 54.063 ms 2024-02-27T06:29:15.862 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 1 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:29:15.863 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-02-27T06:29:15.864 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 5 2024-02-27T06:29:15.864 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:29:15.864 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:29:16.036 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:15 smithi131 ceph-mon[66259]: pgmap v101: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 341 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.3 MiB/s wr, 656 op/s; 185/1764 objects degraded (10.488%) 2024-02-27T06:29:16.036 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:15 smithi131 ceph-mon[66259]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:16.036 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:15 smithi131 ceph-mon[66259]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-02-27T06:29:16.036 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:15 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:16.036 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:15 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:16.036 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:15 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:29:16.036 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:15 smithi131 ceph-mon[66259]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:16.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:15 smithi150 ceph-mon[52803]: pgmap v101: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 341 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.3 MiB/s wr, 656 op/s; 185/1764 objects degraded (10.488%) 2024-02-27T06:29:16.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:15 smithi150 ceph-mon[52803]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:16.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:15 smithi150 ceph-mon[52803]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-02-27T06:29:16.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:15 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:16.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:15 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:16.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:15 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:29:16.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:15 smithi150 ceph-mon[52803]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:16.341 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:29:16.341 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:29:16.341 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:29:16.341 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:29:16.341 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:29:16.341 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:29:16.341 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:29:16.341 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:29:16.342 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:29:16.342 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "7/23 daemons upgraded", 2024-02-27T06:29:16.342 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T06:29:16.342 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:29:16.342 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:29:16.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 218784 10.94 MB/sec execute 354 sec latency 48.373 ms 2024-02-27T06:29:16.970 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 183/1740 objects degraded (10.517%), 23 pgs degraded 2024-02-27T06:29:16.970 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-27T06:29:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: osd.1 (root=default,host=smithi131) is down 2024-02-27T06:29:16.970 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 183/1740 objects degraded (10.517%), 23 pgs degraded 2024-02-27T06:29:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-02-27T06:29:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-02-27T06:29:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.4 is active+undersized+degraded, acting [0,7] 2024-02-27T06:29:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.6 is active+undersized+degraded, acting [6,4] 2024-02-27T06:29:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.8 is active+undersized+degraded, acting [7,0] 2024-02-27T06:29:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-02-27T06:29:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-02-27T06:29:16.970 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.d is active+undersized+degraded, acting [4,3] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.11 is active+undersized+degraded, acting [6,4] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-02-27T06:29:16.971 INFO:teuthology.orchestra.run.smithi131.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-02-27T06:29:17.045 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:16 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/892085119' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:29:17.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:16 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/892085119' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:29:17.542 INFO:teuthology.orchestra.run.smithi131.stdout: 5 219290 10.96 MB/sec execute 355 sec latency 30.677 ms 2024-02-27T06:29:18.054 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:17 smithi131 ceph-mon[66259]: from='client.44185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:18.055 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:17 smithi131 ceph-mon[66259]: pgmap v102: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 341 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.6 MiB/s wr, 559 op/s; 183/1740 objects degraded (10.517%) 2024-02-27T06:29:18.055 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:17 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/3777238967' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:29:18.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:17 smithi150 ceph-mon[52803]: from='client.44185 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:18.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:17 smithi150 ceph-mon[52803]: pgmap v102: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 341 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.6 MiB/s wr, 559 op/s; 183/1740 objects degraded (10.517%) 2024-02-27T06:29:18.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:17 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/3777238967' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:29:18.543 INFO:teuthology.orchestra.run.smithi131.stdout: 5 219730 10.97 MB/sec execute 356 sec latency 38.657 ms 2024-02-27T06:29:19.062 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:18 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 182/1707 objects degraded (10.662%), 23 pgs degraded (PG_DEGRADED) 2024-02-27T06:29:19.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:18 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 182/1707 objects degraded (10.662%), 23 pgs degraded (PG_DEGRADED) 2024-02-27T06:29:19.543 INFO:teuthology.orchestra.run.smithi131.stdout: 5 220092 10.96 MB/sec execute 357 sec latency 50.534 ms 2024-02-27T06:29:20.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:19 smithi131 ceph-mon[66259]: pgmap v103: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 334 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.6 MiB/s wr, 537 op/s; 182/1707 objects degraded (10.662%) 2024-02-27T06:29:20.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:19 smithi150 ceph-mon[52803]: pgmap v103: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 334 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 5.6 MiB/s wr, 537 op/s; 182/1707 objects degraded (10.662%) 2024-02-27T06:29:20.543 INFO:teuthology.orchestra.run.smithi131.stdout: 5 220357 10.94 MB/sec execute 358 sec latency 41.585 ms 2024-02-27T06:29:21.543 INFO:teuthology.orchestra.run.smithi131.stdout: 5 220654 10.94 MB/sec execute 359 sec latency 41.101 ms 2024-02-27T06:29:22.139 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:21 smithi131 ceph-mon[66259]: pgmap v104: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 334 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 479 op/s; 182/1707 objects degraded (10.662%) 2024-02-27T06:29:22.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:21 smithi150 ceph-mon[52803]: pgmap v104: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 334 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.0 MiB/s wr, 479 op/s; 182/1707 objects degraded (10.662%) 2024-02-27T06:29:22.543 INFO:teuthology.orchestra.run.smithi131.stdout: 5 220998 10.93 MB/sec execute 360 sec latency 47.993 ms 2024-02-27T06:29:22.900 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:22 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[75036]: 2024-02-27T06:29:22.721+0000 7f375d75c7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-02-27T06:29:23.543 INFO:teuthology.orchestra.run.smithi131.stdout: 5 221345 10.92 MB/sec execute 361 sec latency 62.788 ms 2024-02-27T06:29:23.938 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:23 smithi131 ceph-mon[66259]: pgmap v105: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 334 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.3 MiB/s wr, 491 op/s; 184/1713 objects degraded (10.741%) 2024-02-27T06:29:24.189 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:24 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[75036]: 2024-02-27T06:29:24.066+0000 7f375d75c7c0 -1 osd.1 85 log_to_monitors true 2024-02-27T06:29:24.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:23 smithi150 ceph-mon[52803]: pgmap v105: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 334 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 5.3 MiB/s wr, 491 op/s; 184/1713 objects degraded (10.741%) 2024-02-27T06:29:24.543 INFO:teuthology.orchestra.run.smithi131.stdout: 5 221677 10.91 MB/sec execute 362 sec latency 48.480 ms 2024-02-27T06:29:24.983 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:24 smithi131 ceph-mon[66259]: from='osd.1 [v2:172.21.15.131:6810/1373671746,v1:172.21.15.131:6811/1373671746]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T06:29:24.984 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:24 smithi131 ceph-mon[66259]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T06:29:24.984 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:24 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:24.984 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:24 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:25.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:24 smithi150 ceph-mon[52803]: from='osd.1 [v2:172.21.15.131:6810/1373671746,v1:172.21.15.131:6811/1373671746]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T06:29:25.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:24 smithi150 ceph-mon[52803]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-27T06:29:25.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:24 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:25.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:24 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:25.503 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:29:25 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[75036]: 2024-02-27T06:29:25.425+0000 7f374abac700 -1 osd.1 85 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:29:25.543 INFO:teuthology.orchestra.run.smithi131.stdout: 5 221944 10.91 MB/sec execute 363 sec latency 121.362 ms 2024-02-27T06:29:26.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:25 smithi131 ceph-mon[66259]: pgmap v106: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 334 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.3 MiB/s wr, 593 op/s; 183/1638 objects degraded (11.172%) 2024-02-27T06:29:26.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:25 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 183/1638 objects degraded (11.172%), 23 pgs degraded (PG_DEGRADED) 2024-02-27T06:29:26.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:25 smithi131 ceph-mon[66259]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-27T06:29:26.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:25 smithi131 ceph-mon[66259]: osdmap e88: 8 total, 7 up, 8 in 2024-02-27T06:29:26.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:25 smithi131 ceph-mon[66259]: from='osd.1 [v2:172.21.15.131:6810/1373671746,v1:172.21.15.131:6811/1373671746]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:29:26.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:25 smithi131 ceph-mon[66259]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:29:26.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:25 smithi131 ceph-mon[66259]: from='osd.1 ' entity='osd.1' 2024-02-27T06:29:26.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:25 smithi150 ceph-mon[52803]: pgmap v106: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 334 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 10 KiB/s rd, 6.3 MiB/s wr, 593 op/s; 183/1638 objects degraded (11.172%) 2024-02-27T06:29:26.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:25 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 183/1638 objects degraded (11.172%), 23 pgs degraded (PG_DEGRADED) 2024-02-27T06:29:26.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:25 smithi150 ceph-mon[52803]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-27T06:29:26.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:25 smithi150 ceph-mon[52803]: osdmap e88: 8 total, 7 up, 8 in 2024-02-27T06:29:26.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:25 smithi150 ceph-mon[52803]: from='osd.1 [v2:172.21.15.131:6810/1373671746,v1:172.21.15.131:6811/1373671746]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:29:26.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:25 smithi150 ceph-mon[52803]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:29:26.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:25 smithi150 ceph-mon[52803]: from='osd.1 ' entity='osd.1' 2024-02-27T06:29:26.543 INFO:teuthology.orchestra.run.smithi131.stdout: 5 222276 10.90 MB/sec execute 364 sec latency 106.272 ms 2024-02-27T06:29:27.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:27.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:27.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:27 smithi131 ceph-mon[66259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:29:27.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:27 smithi131 ceph-mon[66259]: osd.1 [v2:172.21.15.131:6810/1373671746,v1:172.21.15.131:6811/1373671746] boot 2024-02-27T06:29:27.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:27 smithi131 ceph-mon[66259]: osdmap e89: 8 total, 8 up, 8 in 2024-02-27T06:29:27.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:29:27.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:27.348 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:29:27.543 INFO:teuthology.orchestra.run.smithi131.stdout: 5 222394 10.88 MB/sec execute 365 sec latency 1106.309 ms 2024-02-27T06:29:27.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:27.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:27.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:27 smithi150 ceph-mon[52803]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:29:27.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:27 smithi150 ceph-mon[52803]: osd.1 [v2:172.21.15.131:6810/1373671746,v1:172.21.15.131:6811/1373671746] boot 2024-02-27T06:29:27.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:27 smithi150 ceph-mon[52803]: osdmap e89: 8 total, 8 up, 8 in 2024-02-27T06:29:27.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-27T06:29:27.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:27.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:29:28.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:28 smithi131 ceph-mon[66259]: pgmap v109: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.6 MiB/s wr, 348 op/s; 183/1665 objects degraded (10.991%) 2024-02-27T06:29:28.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:28 smithi131 ceph-mon[66259]: osdmap e90: 8 total, 8 up, 8 in 2024-02-27T06:29:28.544 INFO:teuthology.orchestra.run.smithi131.stdout: 5 222412 10.85 MB/sec execute 366 sec latency 2103.395 ms 2024-02-27T06:29:28.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:28 smithi150 ceph-mon[52803]: pgmap v109: 97 pgs: 23 active+undersized+degraded, 16 active+undersized, 58 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.6 MiB/s wr, 348 op/s; 183/1665 objects degraded (10.991%) 2024-02-27T06:29:28.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:28 smithi150 ceph-mon[52803]: osdmap e90: 8 total, 8 up, 8 in 2024-02-27T06:29:29.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:29 smithi131 ceph-mon[66259]: osdmap e91: 8 total, 8 up, 8 in 2024-02-27T06:29:29.456 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:29 smithi131 ceph-mon[66259]: pgmap v112: 97 pgs: 72 active+clean, 6 active+undersized, 8 active+undersized+degraded, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 6 remapped+peering, 3 activating+degraded; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 509 op/s; 119/1659 objects degraded (7.173%); 0 B/s, 0 objects/s recovering 2024-02-27T06:29:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: 5 222767 10.86 MB/sec execute 367 sec latency 2108.922 ms 2024-02-27T06:29:29.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:29 smithi150 ceph-mon[52803]: osdmap e91: 8 total, 8 up, 8 in 2024-02-27T06:29:29.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:29 smithi150 ceph-mon[52803]: pgmap v112: 97 pgs: 72 active+clean, 6 active+undersized, 8 active+undersized+degraded, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 6 remapped+peering, 3 activating+degraded; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.1 MiB/s wr, 509 op/s; 119/1659 objects degraded (7.173%); 0 B/s, 0 objects/s recovering 2024-02-27T06:29:30.475 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:30 smithi131 ceph-mon[66259]: osdmap e92: 8 total, 8 up, 8 in 2024-02-27T06:29:30.544 INFO:teuthology.orchestra.run.smithi131.stdout: 5 223030 10.86 MB/sec execute 368 sec latency 58.676 ms 2024-02-27T06:29:30.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:30 smithi150 ceph-mon[52803]: osdmap e92: 8 total, 8 up, 8 in 2024-02-27T06:29:31.504 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: osdmap e93: 8 total, 8 up, 8 in 2024-02-27T06:29:31.504 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: pgmap v115: 97 pgs: 1 peering, 82 active+clean, 2 active+undersized, 1 active+undersized+degraded, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 6 remapped+peering, 3 activating+degraded; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.2 MiB/s wr, 464 op/s; 89/1650 objects degraded (5.394%); 22 B/s, 61 keys/s, 5 objects/s recovering 2024-02-27T06:29:31.504 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 119/1659 objects degraded (7.173%), 13 pgs degraded (PG_DEGRADED) 2024-02-27T06:29:31.504 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:31.504 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:31.504 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:29:31.504 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:29:31.504 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:31.505 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:31.505 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:29:31.505 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:29:31.505 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:29:31.505 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:29:31.505 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T06:29:31.505 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T06:29:31.505 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-02-27T06:29:31.505 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:31 smithi131 ceph-mon[66259]: osdmap e94: 8 total, 8 up, 8 in 2024-02-27T06:29:31.544 INFO:teuthology.orchestra.run.smithi131.stdout: 5 223172 10.83 MB/sec execute 369 sec latency 903.603 ms 2024-02-27T06:29:31.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: osdmap e93: 8 total, 8 up, 8 in 2024-02-27T06:29:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: pgmap v115: 97 pgs: 1 peering, 82 active+clean, 2 active+undersized, 1 active+undersized+degraded, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 6 remapped+peering, 3 activating+degraded; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.2 MiB/s wr, 464 op/s; 89/1650 objects degraded (5.394%); 22 B/s, 61 keys/s, 5 objects/s recovering 2024-02-27T06:29:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 119/1659 objects degraded (7.173%), 13 pgs degraded (PG_DEGRADED) 2024-02-27T06:29:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:29:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:29:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:29:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:29:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:29:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:29:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T06:29:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T06:29:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-02-27T06:29:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:31 smithi150 ceph-mon[52803]: osdmap e94: 8 total, 8 up, 8 in 2024-02-27T06:29:32.544 INFO:teuthology.orchestra.run.smithi131.stdout: 5 223416 10.81 MB/sec execute 370 sec latency 80.843 ms 2024-02-27T06:29:33.544 INFO:teuthology.orchestra.run.smithi131.stdout: 5 223651 10.80 MB/sec execute 371 sec latency 53.760 ms 2024-02-27T06:29:33.593 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:33 smithi131 ceph-mon[66259]: pgmap v117: 97 pgs: 2 peering, 84 active+clean, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 6 remapped+peering, 3 activating+degraded; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 574 KiB/s wr, 26 op/s; 74/1626 objects degraded (4.551%); 236 KiB/s, 61 keys/s, 11 objects/s recovering 2024-02-27T06:29:33.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:33 smithi150 ceph-mon[52803]: pgmap v117: 97 pgs: 2 peering, 84 active+clean, 1 active+recovery_wait+degraded, 1 active+recovering+degraded, 6 remapped+peering, 3 activating+degraded; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 574 KiB/s wr, 26 op/s; 74/1626 objects degraded (4.551%); 236 KiB/s, 61 keys/s, 11 objects/s recovering 2024-02-27T06:29:34.544 INFO:teuthology.orchestra.run.smithi131.stdout: 5 223851 10.78 MB/sec execute 372 sec latency 65.136 ms 2024-02-27T06:29:34.611 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:34 smithi131 ceph-mon[66259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 74/1626 objects degraded (4.551%), 5 pgs degraded) 2024-02-27T06:29:34.612 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:34 smithi131 ceph-mon[66259]: Cluster is now healthy 2024-02-27T06:29:34.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:34 smithi150 ceph-mon[52803]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 74/1626 objects degraded (4.551%), 5 pgs degraded) 2024-02-27T06:29:34.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:34 smithi150 ceph-mon[52803]: Cluster is now healthy 2024-02-27T06:29:35.544 INFO:teuthology.orchestra.run.smithi131.stdout: 5 224071 10.77 MB/sec execute 373 sec latency 84.814 ms 2024-02-27T06:29:35.632 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:35 smithi131 ceph-mon[66259]: pgmap v118: 97 pgs: 2 peering, 93 active+clean, 2 remapped+peering; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.4 MiB/s wr, 253 op/s; 841 KiB/s, 40 keys/s, 34 objects/s recovering 2024-02-27T06:29:35.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:35 smithi150 ceph-mon[52803]: pgmap v118: 97 pgs: 2 peering, 93 active+clean, 2 remapped+peering; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.4 MiB/s wr, 253 op/s; 841 KiB/s, 40 keys/s, 34 objects/s recovering 2024-02-27T06:29:36.544 INFO:teuthology.orchestra.run.smithi131.stdout: 5 224493 10.77 MB/sec execute 374 sec latency 46.951 ms 2024-02-27T06:29:37.544 INFO:teuthology.orchestra.run.smithi131.stdout: 5 224973 10.78 MB/sec execute 375 sec latency 36.132 ms 2024-02-27T06:29:37.707 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:37 smithi131 ceph-mon[66259]: pgmap v119: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.6 MiB/s wr, 242 op/s; 1.3 MiB/s, 35 keys/s, 37 objects/s recovering 2024-02-27T06:29:37.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:37 smithi150 ceph-mon[52803]: pgmap v119: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.6 MiB/s wr, 242 op/s; 1.3 MiB/s, 35 keys/s, 37 objects/s recovering 2024-02-27T06:29:38.544 INFO:teuthology.orchestra.run.smithi131.stdout: 5 225380 10.77 MB/sec execute 376 sec latency 35.524 ms 2024-02-27T06:29:39.549 INFO:teuthology.orchestra.run.smithi131.stdout: 5 225879 10.79 MB/sec execute 377 sec latency 39.538 ms 2024-02-27T06:29:39.730 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:39 smithi131 ceph-mon[66259]: pgmap v120: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.0 MiB/s wr, 374 op/s; 1.1 MiB/s, 30 objects/s recovering 2024-02-27T06:29:39.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:39 smithi150 ceph-mon[52803]: pgmap v120: 97 pgs: 97 active+clean; 332 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.0 MiB/s wr, 374 op/s; 1.1 MiB/s, 30 objects/s recovering 2024-02-27T06:29:40.549 INFO:teuthology.orchestra.run.smithi131.stdout: 5 226341 10.79 MB/sec execute 378 sec latency 34.611 ms 2024-02-27T06:29:41.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 226703 10.79 MB/sec execute 379 sec latency 48.290 ms 2024-02-27T06:29:41.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:41 smithi131 ceph-mon[66259]: pgmap v121: 97 pgs: 97 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 426 op/s; 914 KiB/s, 24 objects/s recovering 2024-02-27T06:29:41.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:41.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:29:41.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:41 smithi150 ceph-mon[52803]: pgmap v121: 97 pgs: 97 active+clean; 338 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 426 op/s; 914 KiB/s, 24 objects/s recovering 2024-02-27T06:29:41.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:41.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:29:42.549 INFO:teuthology.orchestra.run.smithi131.stdout: 5 227077 10.79 MB/sec execute 380 sec latency 46.321 ms 2024-02-27T06:29:42.804 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:42.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:43.549 INFO:teuthology.orchestra.run.smithi131.stdout: 5 227459 10.79 MB/sec execute 381 sec latency 43.772 ms 2024-02-27T06:29:43.844 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:43 smithi131 ceph-mon[66259]: pgmap v122: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.0 MiB/s wr, 403 op/s; 744 KiB/s, 19 objects/s recovering 2024-02-27T06:29:43.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:43 smithi150 ceph-mon[52803]: pgmap v122: 97 pgs: 97 active+clean; 338 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.0 MiB/s wr, 403 op/s; 744 KiB/s, 19 objects/s recovering 2024-02-27T06:29:44.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 227686 10.77 MB/sec execute 382 sec latency 67.878 ms 2024-02-27T06:29:45.549 INFO:teuthology.orchestra.run.smithi131.stdout: 5 227951 10.76 MB/sec execute 383 sec latency 54.011 ms 2024-02-27T06:29:45.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:45 smithi131 ceph-mon[66259]: pgmap v123: 97 pgs: 97 active+clean; 336 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 524 op/s; 682 KiB/s, 17 objects/s recovering 2024-02-27T06:29:45.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:45 smithi150 ceph-mon[52803]: pgmap v123: 97 pgs: 97 active+clean; 336 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 524 op/s; 682 KiB/s, 17 objects/s recovering 2024-02-27T06:29:46.549 INFO:teuthology.orchestra.run.smithi131.stdout: 5 228183 10.75 MB/sec execute 384 sec latency 90.116 ms 2024-02-27T06:29:46.702 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:46 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T06:29:46.702 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:46 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T06:29:46.702 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:46 smithi131 ceph-mon[66259]: Upgrade: osd.2 is safe to restart 2024-02-27T06:29:46.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:46 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T06:29:46.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:46 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-27T06:29:46.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:46 smithi150 ceph-mon[52803]: Upgrade: osd.2 is safe to restart 2024-02-27T06:29:47.549 INFO:teuthology.orchestra.run.smithi131.stdout: 5 228469 10.74 MB/sec execute 385 sec latency 67.082 ms 2024-02-27T06:29:47.576 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:29:47.753 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:47 smithi131 ceph-mon[66259]: pgmap v124: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.2 MiB/s wr, 514 op/s; 341 KiB/s, 4 objects/s recovering 2024-02-27T06:29:47.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:47 smithi150 ceph-mon[52803]: pgmap v124: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 5.2 MiB/s wr, 514 op/s; 341 KiB/s, 4 objects/s recovering 2024-02-27T06:29:48.335 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:29:48.335 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (5m) 23s ago 14m 22.0M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:29:48.335 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (2m) 23s ago 14m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:29:48.335 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (116s) 111s ago 14m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:29:48.335 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (5m) 23s ago 14m 54.0M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:29:48.335 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (8m) 23s ago 8m 38.7M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:29:48.335 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (8m) 111s ago 8m 14.2M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:29:48.335 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (4m) 23s ago 15m 501M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:29:48.335 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (3m) 111s ago 14m 449M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:29:48.335 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (3m) 23s ago 16m 47.2M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:29:48.335 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (3m) 111s ago 14m 36.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:29:48.335 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (8m) 23s ago 8m 260M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:29:48.336 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (8m) 111s ago 8m 81.4M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:29:48.336 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (5m) 23s ago 14m 21.5M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:29:48.336 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (5m) 111s ago 13m 17.2M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:29:48.336 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (83s) 23s ago 13m 258M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:29:48.336 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (34s) 23s ago 12m 12.4M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:29:48.336 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (12m) 23s ago 12m 541M 1936M 17.2.0 e1d6a67b021e ab37f0eca956 2024-02-27T06:29:48.336 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (11m) 23s ago 11m 399M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:29:48.336 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (11m) 111s ago 11m 557M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:29:48.336 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (11m) 111s ago 11m 436M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:29:48.336 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (10m) 111s ago 10m 347M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:29:48.336 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (10m) 111s ago 10m 542M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:29:48.336 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (4m) 23s ago 13m 67.9M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:29:48.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 228695 10.73 MB/sec execute 386 sec latency 76.420 ms 2024-02-27T06:29:48.799 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:48 smithi131 ceph-mon[66259]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:48.799 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:48 smithi131 ceph-mon[66259]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:48.799 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:48.799 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T06:29:48.799 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:29:48.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:48 smithi150 ceph-mon[52803]: from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:48.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:48 smithi150 ceph-mon[52803]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:48.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:48.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-27T06:29:48.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:29:48.903 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:29:48.903 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:29:48.904 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-02-27T06:29:48.905 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 6 2024-02-27T06:29:48.905 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:29:48.905 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:29:49.374 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:29:49.374 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:29:49.374 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:29:49.374 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:29:49.374 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:29:49.374 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:29:49.374 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:29:49.374 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:29:49.374 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:29:49.374 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "8/23 daemons upgraded", 2024-02-27T06:29:49.374 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T06:29:49.375 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:29:49.375 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:29:49.549 INFO:teuthology.orchestra.run.smithi131.stdout: 5 228956 10.72 MB/sec execute 387 sec latency 49.347 ms 2024-02-27T06:29:49.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:49 smithi131 ceph-mon[66259]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:49.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:49 smithi131 ceph-mon[66259]: pgmap v125: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.8 MiB/s wr, 614 op/s 2024-02-27T06:29:49.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:49 smithi131 ceph-mon[66259]: Upgrade: Updating osd.2 2024-02-27T06:29:49.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:49 smithi131 ceph-mon[66259]: Deploying daemon osd.2 on smithi131 2024-02-27T06:29:49.839 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:49 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/1543302362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:29:49.945 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:29:50.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:49 smithi150 ceph-mon[52803]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:50.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:49 smithi150 ceph-mon[52803]: pgmap v125: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.8 MiB/s wr, 614 op/s 2024-02-27T06:29:50.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:49 smithi150 ceph-mon[52803]: Upgrade: Updating osd.2 2024-02-27T06:29:50.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:49 smithi150 ceph-mon[52803]: Deploying daemon osd.2 on smithi131 2024-02-27T06:29:50.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:49 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/1543302362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:29:50.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 229256 10.72 MB/sec execute 388 sec latency 52.319 ms 2024-02-27T06:29:50.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:50 smithi131 ceph-mon[66259]: from='client.34294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:50.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:50 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/3282232527' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:29:51.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:50 smithi150 ceph-mon[52803]: from='client.34294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:29:51.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:50 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/3282232527' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:29:51.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 229597 10.71 MB/sec execute 389 sec latency 50.778 ms 2024-02-27T06:29:51.627 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:51 smithi131 ceph-mon[66259]: pgmap v126: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.8 MiB/s wr, 510 op/s 2024-02-27T06:29:52.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:51 smithi150 ceph-mon[52803]: pgmap v126: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.8 MiB/s wr, 510 op/s 2024-02-27T06:29:52.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 229899 10.71 MB/sec execute 390 sec latency 42.695 ms 2024-02-27T06:29:53.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 230127 10.70 MB/sec execute 391 sec latency 50.206 ms 2024-02-27T06:29:53.698 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:53 smithi131 ceph-mon[66259]: pgmap v127: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.4 MiB/s wr, 414 op/s 2024-02-27T06:29:53.949 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:53 smithi131 systemd[1]: Stopping Ceph osd.2 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:29:54.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:53 smithi150 ceph-mon[52803]: pgmap v127: 97 pgs: 97 active+clean; 334 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.4 MiB/s wr, 414 op/s 2024-02-27T06:29:54.453 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:54 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2[42985]: 2024-02-27T06:29:54.276+0000 7f8849bba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:29:54.453 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:54 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2[42985]: 2024-02-27T06:29:54.276+0000 7f8849bba700 -1 osd.2 94 *** Got signal Terminated *** 2024-02-27T06:29:54.453 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:54 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2[42985]: 2024-02-27T06:29:54.276+0000 7f8849bba700 -1 osd.2 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:29:54.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 230357 10.69 MB/sec execute 392 sec latency 77.717 ms 2024-02-27T06:29:54.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:54 smithi131 ceph-mon[66259]: osd.2 marked itself down and dead 2024-02-27T06:29:55.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:54 smithi150 ceph-mon[52803]: osd.2 marked itself down and dead 2024-02-27T06:29:55.541 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:55 smithi131 podman[77272]: 2024-02-27 06:29:55.274915834 +0000 UTC m=+1.248832981 container died ab37f0eca956f6eeb692342dd542bccb55f75b78d6d24961e286c7fb9bf808e1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.openshift.expose-services=, com.redhat.component=centos-stream-container, distribution-scope=public, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, version=8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, name=centos-stream, ceph=True, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git) 2024-02-27T06:29:55.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 230388 10.67 MB/sec execute 393 sec latency 931.515 ms 2024-02-27T06:29:55.794 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:55 smithi131 ceph-mon[66259]: pgmap v128: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.4 MiB/s wr, 550 op/s 2024-02-27T06:29:55.794 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:55 smithi131 ceph-mon[66259]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:29:55.794 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:55 smithi131 ceph-mon[66259]: osdmap e95: 8 total, 7 up, 8 in 2024-02-27T06:29:56.053 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:55 smithi131 podman[77272]: 2024-02-27 06:29:55.958618287 +0000 UTC m=+1.932535445 container cleanup ab37f0eca956f6eeb692342dd542bccb55f75b78d6d24961e286c7fb9bf808e1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.tags=base centos centos-stream, release=754, io.buildah.version=1.19.8, GIT_CLEAN=True, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-27T06:29:56.053 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:55 smithi131 bash[77272]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2 2024-02-27T06:29:56.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:55 smithi150 ceph-mon[52803]: pgmap v128: 97 pgs: 97 active+clean; 339 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 5.4 MiB/s wr, 550 op/s 2024-02-27T06:29:56.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:55 smithi150 ceph-mon[52803]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:29:56.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:55 smithi150 ceph-mon[52803]: osdmap e95: 8 total, 7 up, 8 in 2024-02-27T06:29:56.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 230655 10.65 MB/sec execute 394 sec latency 1044.758 ms 2024-02-27T06:29:56.557 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:56 smithi131 podman[77287]: 2024-02-27 06:29:56.387232329 +0000 UTC m=+1.107428629 container remove ab37f0eca956f6eeb692342dd542bccb55f75b78d6d24961e286c7fb9bf808e1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, release=754, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-02-27T06:29:56.819 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:56 smithi131 ceph-mon[66259]: osdmap e96: 8 total, 7 up, 8 in 2024-02-27T06:29:56.819 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:56 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:56.819 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:56 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:29:57.078 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:56 smithi131 podman[77376]: 2024-02-27 06:29:56.807060849 +0000 UTC m=+0.034563136 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:29:57.078 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:56 smithi131 podman[77376]: 2024-02-27 06:29:56.980915844 +0000 UTC m=+0.208418124 container create a520bda273de7109ada3b0f739b840bbc0df6e8618b2726dd0ae547fd0e326e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-27T06:29:57.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:56 smithi150 ceph-mon[52803]: osdmap e96: 8 total, 7 up, 8 in 2024-02-27T06:29:57.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:56 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:29:57.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:56 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:29:57.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 230963 10.65 MB/sec execute 395 sec latency 42.407 ms 2024-02-27T06:29:57.599 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:57 smithi131 podman[77376]: 2024-02-27 06:29:57.39375103 +0000 UTC m=+0.621253317 container init a520bda273de7109ada3b0f739b840bbc0df6e8618b2726dd0ae547fd0e326e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) 2024-02-27T06:29:57.599 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:57 smithi131 podman[77376]: 2024-02-27 06:29:57.40175872 +0000 UTC m=+0.629260996 container start a520bda273de7109ada3b0f739b840bbc0df6e8618b2726dd0ae547fd0e326e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-02-27T06:29:57.599 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:57 smithi131 podman[77376]: 2024-02-27 06:29:57.562101968 +0000 UTC m=+0.789604247 container attach a520bda273de7109ada3b0f739b840bbc0df6e8618b2726dd0ae547fd0e326e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:29:57.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:57 smithi131 ceph-mon[66259]: pgmap v131: 97 pgs: 1 active+undersized, 1 active+undersized+degraded, 6 peering, 4 stale+active+clean, 85 active+clean; 336 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.3 MiB/s wr, 286 op/s; 9/1644 objects degraded (0.547%) 2024-02-27T06:29:57.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:57 smithi131 ceph-mon[66259]: Health check failed: Degraded data redundancy: 9/1644 objects degraded (0.547%), 1 pg degraded (PG_DEGRADED) 2024-02-27T06:29:58.107 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:57 smithi131 podman[77376]: 2024-02-27 06:29:57.848396628 +0000 UTC m=+1.075898904 container died a520bda273de7109ada3b0f739b840bbc0df6e8618b2726dd0ae547fd0e326e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:29:58.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:57 smithi150 ceph-mon[52803]: pgmap v131: 97 pgs: 1 active+undersized, 1 active+undersized+degraded, 6 peering, 4 stale+active+clean, 85 active+clean; 336 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.3 MiB/s wr, 286 op/s; 9/1644 objects degraded (0.547%) 2024-02-27T06:29:58.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:57 smithi150 ceph-mon[52803]: Health check failed: Degraded data redundancy: 9/1644 objects degraded (0.547%), 1 pg degraded (PG_DEGRADED) 2024-02-27T06:29:58.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 231364 10.64 MB/sec execute 396 sec latency 36.430 ms 2024-02-27T06:29:59.122 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:59 smithi131 podman[77401]: 2024-02-27 06:29:59.04669494 +0000 UTC m=+1.191659513 container remove a520bda273de7109ada3b0f739b840bbc0df6e8618b2726dd0ae547fd0e326e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:29:59.122 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:59 smithi131 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-27T06:29:59.122 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:59 smithi131 systemd[1]: Stopped Ceph osd.2 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:29:59.122 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:59 smithi131 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.2.service: Consumed 3min 7.485s CPU time. 2024-02-27T06:29:59.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 231713 10.64 MB/sec execute 397 sec latency 43.072 ms 2024-02-27T06:29:59.635 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:59 smithi131 systemd[1]: Starting Ceph osd.2 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:29:59.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:29:59 smithi150 ceph-mon[52803]: pgmap v132: 97 pgs: 4 active+undersized, 17 active+undersized+degraded, 6 peering, 70 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.5 MiB/s wr, 446 op/s; 111/1602 objects degraded (6.929%) 2024-02-27T06:29:59.894 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:29:59 smithi131 ceph-mon[66259]: pgmap v132: 97 pgs: 4 active+undersized, 17 active+undersized+degraded, 6 peering, 70 active+clean; 333 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.5 MiB/s wr, 446 op/s; 111/1602 objects degraded (6.929%) 2024-02-27T06:29:59.894 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:59 smithi131 podman[77496]: 2024-02-27 06:29:59.570426976 +0000 UTC m=+0.026065214 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:29:59.894 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:29:59 smithi131 podman[77496]: 2024-02-27 06:29:59.747353789 +0000 UTC m=+0.202992015 container create f82e6b2e160bcb355d1f47f9e6a1c2c734a72cfa1d26c3921f5a3073038fc8bb (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-02-27T06:30:00.153 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 podman[77496]: 2024-02-27 06:30:00.133812076 +0000 UTC m=+0.589450309 container init f82e6b2e160bcb355d1f47f9e6a1c2c734a72cfa1d26c3921f5a3073038fc8bb (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-02-27T06:30:00.417 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 podman[77496]: 2024-02-27 06:30:00.144600825 +0000 UTC m=+0.600239058 container start f82e6b2e160bcb355d1f47f9e6a1c2c734a72cfa1d26c3921f5a3073038fc8bb (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222) 2024-02-27T06:30:00.417 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 podman[77496]: 2024-02-27 06:30:00.350780523 +0000 UTC m=+0.806418750 container attach f82e6b2e160bcb355d1f47f9e6a1c2c734a72cfa1d26c3921f5a3073038fc8bb (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:30:00.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 232123 10.64 MB/sec execute 398 sec latency 51.723 ms 2024-02-27T06:30:00.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 111/1602 objects degraded (6.929%), 17 pgs degraded 2024-02-27T06:30:00.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: [WRN] OSD_DOWN: 1 osds down 2024-02-27T06:30:00.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: osd.2 (root=default,host=smithi131) is down 2024-02-27T06:30:00.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: [WRN] PG_DEGRADED: Degraded data redundancy: 111/1602 objects degraded (6.929%), 17 pgs degraded 2024-02-27T06:30:00.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-02-27T06:30:00.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 2.7 is active+undersized+degraded, acting [6,7] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 2.c is active+undersized+degraded, acting [5,0] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 2.e is active+undersized+degraded, acting [3,7] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 2.f is active+undersized+degraded, acting [4,0] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 2.16 is active+undersized+degraded, acting [5,6] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 2.1c is active+undersized+degraded, acting [7,5] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-02-27T06:30:00.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 4.a is active+undersized+degraded, acting [6,7] 2024-02-27T06:30:00.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-mon[66259]: pg 4.19 is active+undersized+degraded, acting [3,0] 2024-02-27T06:30:00.923 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate[77511]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-27T06:30:00.923 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate[77511]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-27T06:30:00.923 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 bash[77496]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-27T06:30:00.923 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 bash[77496]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-27T06:30:00.923 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 bash[77496]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-27T06:30:00.923 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 bash[77496]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-27T06:30:00.923 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 bash[77496]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-27T06:30:00.923 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 bash[77496]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-27T06:30:00.923 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 bash[77496]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-27T06:30:00.924 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate[77511]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-27T06:30:00.924 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate[77511]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-27T06:30:00.924 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate[77511]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-27T06:30:00.924 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate[77511]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-27T06:30:00.924 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate[77511]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-27T06:30:00.924 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 conmon[77511]: conmon f82e6b2e160bcb355d1f : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-f82e6b2e160bcb355d1f47f9e6a1c2c734a72cfa1d26c3921f5a3073038fc8bb.scope/container/memory.events 2024-02-27T06:30:00.924 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:00 smithi131 podman[77496]: 2024-02-27 06:30:00.844832753 +0000 UTC m=+1.300470980 container died f82e6b2e160bcb355d1f47f9e6a1c2c734a72cfa1d26c3921f5a3073038fc8bb (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:30:01.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 111/1602 objects degraded (6.929%), 17 pgs degraded 2024-02-27T06:30:01.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: [WRN] OSD_DOWN: 1 osds down 2024-02-27T06:30:01.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: osd.2 (root=default,host=smithi131) is down 2024-02-27T06:30:01.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: [WRN] PG_DEGRADED: Degraded data redundancy: 111/1602 objects degraded (6.929%), 17 pgs degraded 2024-02-27T06:30:01.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-02-27T06:30:01.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-02-27T06:30:01.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 2.7 is active+undersized+degraded, acting [6,7] 2024-02-27T06:30:01.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 2.c is active+undersized+degraded, acting [5,0] 2024-02-27T06:30:01.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 2.e is active+undersized+degraded, acting [3,7] 2024-02-27T06:30:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 2.f is active+undersized+degraded, acting [4,0] 2024-02-27T06:30:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 2.16 is active+undersized+degraded, acting [5,6] 2024-02-27T06:30:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-02-27T06:30:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 2.1c is active+undersized+degraded, acting [7,5] 2024-02-27T06:30:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-02-27T06:30:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-02-27T06:30:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-02-27T06:30:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-02-27T06:30:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-02-27T06:30:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-02-27T06:30:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 4.a is active+undersized+degraded, acting [6,7] 2024-02-27T06:30:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:00 smithi150 ceph-mon[52803]: pg 4.19 is active+undersized+degraded, acting [3,0] 2024-02-27T06:30:01.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 232425 10.63 MB/sec execute 399 sec latency 53.945 ms 2024-02-27T06:30:01.688 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:01 smithi131 ceph-mon[66259]: pgmap v133: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.0 MiB/s wr, 565 op/s; 151/1611 objects degraded (9.373%) 2024-02-27T06:30:02.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:01 smithi150 ceph-mon[52803]: pgmap v133: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.0 MiB/s wr, 565 op/s; 151/1611 objects degraded (9.373%) 2024-02-27T06:30:02.200 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:02 smithi131 podman[77675]: 2024-02-27 06:30:02.136933894 +0000 UTC m=+1.285428630 container remove f82e6b2e160bcb355d1f47f9e6a1c2c734a72cfa1d26c3921f5a3073038fc8bb (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:30:02.451 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:02 smithi131 podman[77731]: 2024-02-27 06:30:02.307488086 +0000 UTC m=+0.022222116 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:30:02.451 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:02 smithi131 podman[77731]: 2024-02-27 06:30:02.4248313 +0000 UTC m=+0.139565286 container create 63a1c6f33f95d2374a84449b58e453270b4811af862271ce462cd5d6a1a50876 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-27T06:30:02.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 232795 10.64 MB/sec execute 400 sec latency 54.568 ms 2024-02-27T06:30:02.966 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:02 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 153/1623 objects degraded (9.427%), 20 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:02.966 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:02 smithi131 podman[77731]: 2024-02-27 06:30:02.762254583 +0000 UTC m=+0.476988580 container init 63a1c6f33f95d2374a84449b58e453270b4811af862271ce462cd5d6a1a50876 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222) 2024-02-27T06:30:02.966 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:02 smithi131 podman[77731]: 2024-02-27 06:30:02.775034609 +0000 UTC m=+0.489768609 container start 63a1c6f33f95d2374a84449b58e453270b4811af862271ce462cd5d6a1a50876 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, ceph=True) 2024-02-27T06:30:02.966 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:02 smithi131 ceph-osd[77749]: -- 172.21.15.131:0/1972532791 <== mon.0 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5649a02d9680 con 0x5649a02a3000 2024-02-27T06:30:03.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:02 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 153/1623 objects degraded (9.427%), 20 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:03.221 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:03 smithi131 bash[77731]: 63a1c6f33f95d2374a84449b58e453270b4811af862271ce462cd5d6a1a50876 2024-02-27T06:30:03.221 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:03 smithi131 systemd[1]: Started Ceph osd.2 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:30:03.472 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:03 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2[77745]: 2024-02-27T06:30:03.345+0000 7f8aff8f47c0 -1 Falling back to public interface 2024-02-27T06:30:03.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 233095 10.62 MB/sec execute 401 sec latency 45.000 ms 2024-02-27T06:30:04.488 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:04 smithi131 ceph-mon[66259]: pgmap v134: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.3 MiB/s wr, 351 op/s; 153/1623 objects degraded (9.427%) 2024-02-27T06:30:04.488 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:04 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:04.488 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:04 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:04.488 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:04 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:30:04.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 233429 10.63 MB/sec execute 402 sec latency 38.591 ms 2024-02-27T06:30:04.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:04 smithi150 ceph-mon[52803]: pgmap v134: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 337 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.3 MiB/s wr, 351 op/s; 153/1623 objects degraded (9.427%) 2024-02-27T06:30:04.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:04 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:04.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:04 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:04.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:04 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:30:05.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 233775 10.63 MB/sec execute 403 sec latency 36.643 ms 2024-02-27T06:30:06.280 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:06 smithi131 ceph-mon[66259]: pgmap v135: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 337 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.0 MiB/s wr, 356 op/s; 155/1641 objects degraded (9.445%) 2024-02-27T06:30:06.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 234102 10.62 MB/sec execute 404 sec latency 47.882 ms 2024-02-27T06:30:06.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:06 smithi150 ceph-mon[52803]: pgmap v135: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 337 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.0 MiB/s wr, 356 op/s; 155/1641 objects degraded (9.445%) 2024-02-27T06:30:07.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 234413 10.61 MB/sec execute 405 sec latency 48.200 ms 2024-02-27T06:30:07.608 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:07 smithi131 ceph-mon[66259]: pgmap v136: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 459 op/s; 163/1665 objects degraded (9.790%) 2024-02-27T06:30:07.608 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:07 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:07.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:07 smithi150 ceph-mon[52803]: pgmap v136: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 342 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.6 MiB/s wr, 459 op/s; 163/1665 objects degraded (9.790%) 2024-02-27T06:30:07.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:07 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:08.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 234716 10.60 MB/sec execute 406 sec latency 64.765 ms 2024-02-27T06:30:09.393 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:09 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2[77745]: 2024-02-27T06:30:09.186+0000 7f8aff8f47c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-02-27T06:30:09.550 INFO:teuthology.orchestra.run.smithi131.stdout: 5 235042 10.60 MB/sec execute 407 sec latency 80.296 ms 2024-02-27T06:30:09.643 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:09 smithi131 ceph-mon[66259]: pgmap v137: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.2 MiB/s wr, 546 op/s; 163/1665 objects degraded (9.790%) 2024-02-27T06:30:09.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:09 smithi150 ceph-mon[52803]: pgmap v137: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 340 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.2 MiB/s wr, 546 op/s; 163/1665 objects degraded (9.790%) 2024-02-27T06:30:10.399 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:10 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2[77745]: 2024-02-27T06:30:10.365+0000 7f8aff8f47c0 -1 osd.2 94 log_to_monitors true 2024-02-27T06:30:10.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 235322 10.59 MB/sec execute 408 sec latency 59.498 ms 2024-02-27T06:30:10.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:10 smithi131 ceph-mon[66259]: from='osd.2 [v2:172.21.15.131:6818/3043875388,v1:172.21.15.131:6819/3043875388]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T06:30:10.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:10 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 163/1665 objects degraded (9.790%), 20 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:10.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:10 smithi150 ceph-mon[52803]: from='osd.2 [v2:172.21.15.131:6818/3043875388,v1:172.21.15.131:6819/3043875388]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-27T06:30:10.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:10 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 163/1665 objects degraded (9.790%), 20 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:10.913 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:30:10 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2[77745]: 2024-02-27T06:30:10.644+0000 7f8af0d4c700 -1 osd.2 94 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:30:11.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 235802 10.60 MB/sec execute 409 sec latency 59.282 ms 2024-02-27T06:30:11.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:11 smithi131 ceph-mon[66259]: pgmap v138: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.5 MiB/s wr, 442 op/s; 160/1653 objects degraded (9.679%) 2024-02-27T06:30:11.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:11 smithi131 ceph-mon[66259]: from='osd.2 [v2:172.21.15.131:6818/3043875388,v1:172.21.15.131:6819/3043875388]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-27T06:30:11.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:11 smithi131 ceph-mon[66259]: osdmap e97: 8 total, 7 up, 8 in 2024-02-27T06:30:11.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:11 smithi131 ceph-mon[66259]: from='osd.2 [v2:172.21.15.131:6818/3043875388,v1:172.21.15.131:6819/3043875388]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:30:11.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:11.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:30:12.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:11 smithi150 ceph-mon[52803]: pgmap v138: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.5 MiB/s wr, 442 op/s; 160/1653 objects degraded (9.679%) 2024-02-27T06:30:12.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:11 smithi150 ceph-mon[52803]: from='osd.2 [v2:172.21.15.131:6818/3043875388,v1:172.21.15.131:6819/3043875388]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-27T06:30:12.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:11 smithi150 ceph-mon[52803]: osdmap e97: 8 total, 7 up, 8 in 2024-02-27T06:30:12.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:11 smithi150 ceph-mon[52803]: from='osd.2 [v2:172.21.15.131:6818/3043875388,v1:172.21.15.131:6819/3043875388]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:30:12.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:12.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:30:12.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 235858 10.57 MB/sec execute 410 sec latency 954.313 ms 2024-02-27T06:30:13.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:12 smithi131 ceph-mon[66259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:30:13.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:12 smithi131 ceph-mon[66259]: osd.2 [v2:172.21.15.131:6818/3043875388,v1:172.21.15.131:6819/3043875388] boot 2024-02-27T06:30:13.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:12 smithi131 ceph-mon[66259]: osdmap e98: 8 total, 8 up, 8 in 2024-02-27T06:30:13.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:12 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:30:13.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:12 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:13.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:12 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:13.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:12 smithi131 ceph-mon[66259]: osdmap e99: 8 total, 8 up, 8 in 2024-02-27T06:30:13.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:12 smithi150 ceph-mon[52803]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:30:13.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:12 smithi150 ceph-mon[52803]: osd.2 [v2:172.21.15.131:6818/3043875388,v1:172.21.15.131:6819/3043875388] boot 2024-02-27T06:30:13.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:12 smithi150 ceph-mon[52803]: osdmap e98: 8 total, 8 up, 8 in 2024-02-27T06:30:13.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:12 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-27T06:30:13.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:12 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:13.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:12 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:13.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:12 smithi150 ceph-mon[52803]: osdmap e99: 8 total, 8 up, 8 in 2024-02-27T06:30:13.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 235858 10.55 MB/sec execute 411 sec latency 1954.343 ms 2024-02-27T06:30:14.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:13 smithi131 ceph-mon[66259]: pgmap v141: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.2 MiB/s wr, 441 op/s; 160/1671 objects degraded (9.575%) 2024-02-27T06:30:14.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:13 smithi131 ceph-mon[66259]: osdmap e100: 8 total, 8 up, 8 in 2024-02-27T06:30:14.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:13 smithi150 ceph-mon[52803]: pgmap v141: 97 pgs: 7 active+undersized, 20 active+undersized+degraded, 70 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.2 MiB/s wr, 441 op/s; 160/1671 objects degraded (9.575%) 2024-02-27T06:30:14.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:13 smithi150 ceph-mon[52803]: osdmap e100: 8 total, 8 up, 8 in 2024-02-27T06:30:14.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 235994 10.53 MB/sec execute 412 sec latency 2454.757 ms 2024-02-27T06:30:14.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:14.849 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:15.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:15.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:15.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 236085 10.51 MB/sec execute 413 sec latency 767.027 ms 2024-02-27T06:30:15.759 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:15 smithi131 ceph-mon[66259]: pgmap v144: 97 pgs: 15 peering, 5 active+undersized, 7 active+undersized+degraded, 70 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s wr, 515 op/s; 68/1671 objects degraded (4.069%) 2024-02-27T06:30:15.759 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:15 smithi131 ceph-mon[66259]: osdmap e101: 8 total, 8 up, 8 in 2024-02-27T06:30:15.759 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:15 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 68/1671 objects degraded (4.069%), 7 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:15.759 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:15 smithi131 ceph-mon[66259]: osdmap e102: 8 total, 8 up, 8 in 2024-02-27T06:30:16.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:15 smithi150 ceph-mon[52803]: pgmap v144: 97 pgs: 15 peering, 5 active+undersized, 7 active+undersized+degraded, 70 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s wr, 515 op/s; 68/1671 objects degraded (4.069%) 2024-02-27T06:30:16.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:15 smithi150 ceph-mon[52803]: osdmap e101: 8 total, 8 up, 8 in 2024-02-27T06:30:16.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:15 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 68/1671 objects degraded (4.069%), 7 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:16.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:15 smithi150 ceph-mon[52803]: osdmap e102: 8 total, 8 up, 8 in 2024-02-27T06:30:16.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 236442 10.51 MB/sec execute 414 sec latency 979.036 ms 2024-02-27T06:30:17.542 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:17 smithi131 ceph-mon[66259]: pgmap v147: 97 pgs: 15 peering, 1 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.8 MiB/s wr, 519 op/s; 34/1629 objects degraded (2.087%); 587 KiB/s, 10 objects/s recovering 2024-02-27T06:30:17.542 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:17 smithi131 ceph-mon[66259]: osdmap e103: 8 total, 8 up, 8 in 2024-02-27T06:30:17.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 236702 10.51 MB/sec execute 415 sec latency 821.603 ms 2024-02-27T06:30:17.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:17 smithi150 ceph-mon[52803]: pgmap v147: 97 pgs: 15 peering, 1 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 331 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.8 MiB/s wr, 519 op/s; 34/1629 objects degraded (2.087%); 587 KiB/s, 10 objects/s recovering 2024-02-27T06:30:17.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:17 smithi150 ceph-mon[52803]: osdmap e103: 8 total, 8 up, 8 in 2024-02-27T06:30:18.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 237100 10.51 MB/sec execute 416 sec latency 631.080 ms 2024-02-27T06:30:18.574 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:18 smithi131 ceph-mon[66259]: osdmap e104: 8 total, 8 up, 8 in 2024-02-27T06:30:18.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:18 smithi150 ceph-mon[52803]: osdmap e104: 8 total, 8 up, 8 in 2024-02-27T06:30:19.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 237402 10.50 MB/sec execute 417 sec latency 633.245 ms 2024-02-27T06:30:19.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: pgmap v150: 97 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 2.9 MiB/s wr, 263 op/s; 34/1644 objects degraded (2.068%); 2.0 MiB/s, 37 objects/s recovering 2024-02-27T06:30:19.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: osdmap e105: 8 total, 8 up, 8 in 2024-02-27T06:30:19.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:19.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:19.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:30:19.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:30:19.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:19.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:19.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:30:19.595 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:19.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:19.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:19.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T06:30:19.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T06:30:19.596 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:19 smithi131 ceph-mon[66259]: Upgrade: osd.3 is safe to restart 2024-02-27T06:30:19.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: pgmap v150: 97 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.5 KiB/s rd, 2.9 MiB/s wr, 263 op/s; 34/1644 objects degraded (2.068%); 2.0 MiB/s, 37 objects/s recovering 2024-02-27T06:30:19.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: osdmap e105: 8 total, 8 up, 8 in 2024-02-27T06:30:19.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:19.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:19.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:30:19.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:30:19.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:19.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:19.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:30:19.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:19.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:19.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:19.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T06:30:19.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-27T06:30:19.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:19 smithi150 ceph-mon[52803]: Upgrade: osd.3 is safe to restart 2024-02-27T06:30:20.498 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:30:20.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 237643 10.48 MB/sec execute 418 sec latency 47.511 ms 2024-02-27T06:30:20.605 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:20 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 34/1644 objects degraded (2.068%), 1 pg degraded (PG_DEGRADED) 2024-02-27T06:30:20.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:20 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 34/1644 objects degraded (2.068%), 1 pg degraded (PG_DEGRADED) 2024-02-27T06:30:21.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 237946 10.48 MB/sec execute 419 sec latency 47.035 ms 2024-02-27T06:30:21.560 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:30:21.560 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (6m) 8s ago 15m 22.0M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:30:21.560 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (2m) 8s ago 15m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:30:21.560 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (2m) 2m ago 14m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:30:21.560 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (6m) 8s ago 15m 54.1M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (9m) 8s ago 9m 38.7M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (9m) 2m ago 9m 14.2M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (5m) 8s ago 16m 503M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (4m) 2m ago 14m 449M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (4m) 8s ago 16m 47.3M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (3m) 2m ago 14m 36.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (9m) 8s ago 9m 263M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (9m) 2m ago 9m 81.4M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (5m) 8s ago 15m 17.5M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (5m) 2m ago 14m 17.2M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (116s) 8s ago 13m 312M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (67s) 8s ago 13m 258M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (19s) 8s ago 12m 12.0M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (12m) 8s ago 12m 421M 1936M 17.2.0 e1d6a67b021e a85c70386ff6 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (12m) 2m ago 12m 557M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:30:21.561 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (11m) 2m ago 11m 436M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:30:21.562 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (11m) 2m ago 11m 347M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:30:21.562 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (10m) 2m ago 10m 542M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:30:21.562 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (4m) 8s ago 14m 68.0M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:30:21.630 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:21 smithi131 ceph-mon[66259]: pgmap v152: 97 pgs: 1 peering, 96 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.4 MiB/s wr, 217 op/s; 1.8 MiB/s, 32 objects/s recovering 2024-02-27T06:30:21.630 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:21 smithi131 ceph-mon[66259]: from='client.34306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:21.630 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:21 smithi131 ceph-mon[66259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/1644 objects degraded (2.068%), 1 pg degraded) 2024-02-27T06:30:21.630 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:21 smithi131 ceph-mon[66259]: Cluster is now healthy 2024-02-27T06:30:21.630 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:21 smithi131 ceph-mon[66259]: from='client.44217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:21.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:21 smithi150 ceph-mon[52803]: pgmap v152: 97 pgs: 1 peering, 96 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.4 MiB/s wr, 217 op/s; 1.8 MiB/s, 32 objects/s recovering 2024-02-27T06:30:21.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:21 smithi150 ceph-mon[52803]: from='client.34306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:21.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:21 smithi150 ceph-mon[52803]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/1644 objects degraded (2.068%), 1 pg degraded) 2024-02-27T06:30:21.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:21 smithi150 ceph-mon[52803]: Cluster is now healthy 2024-02-27T06:30:21.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:21 smithi150 ceph-mon[52803]: from='client.44217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:22.193 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:30:22.193 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:30:22.193 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:30:22.193 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:30:22.193 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:30:22.193 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:30:22.193 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:30:22.193 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:30:22.193 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-27T06:30:22.193 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 3 2024-02-27T06:30:22.194 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:30:22.194 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:30:22.194 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:30:22.194 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:30:22.194 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:30:22.194 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-27T06:30:22.194 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 7 2024-02-27T06:30:22.194 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:30:22.194 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:30:22.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 238246 10.48 MB/sec execute 420 sec latency 57.374 ms 2024-02-27T06:30:22.648 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:22 smithi131 ceph-mon[66259]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:22.648 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:22 smithi131 ceph-mon[66259]: Upgrade: Updating osd.3 2024-02-27T06:30:22.648 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:22.648 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T06:30:22.648 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:30:22.648 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:22 smithi131 ceph-mon[66259]: Deploying daemon osd.3 on smithi131 2024-02-27T06:30:22.648 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:22 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/1447941476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:22.741 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:30:22.742 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:30:22.742 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:30:22.742 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:30:22.742 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:30:22.742 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:30:22.742 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:30:22.742 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:30:22.742 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:30:22.742 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "9/23 daemons upgraded", 2024-02-27T06:30:22.742 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T06:30:22.742 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:30:22.742 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:30:22.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:22 smithi150 ceph-mon[52803]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:22.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:22 smithi150 ceph-mon[52803]: Upgrade: Updating osd.3 2024-02-27T06:30:22.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:22.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-27T06:30:22.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:30:22.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:22 smithi150 ceph-mon[52803]: Deploying daemon osd.3 on smithi131 2024-02-27T06:30:22.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:22 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/1447941476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:23.362 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:30:23.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 238601 10.47 MB/sec execute 421 sec latency 43.081 ms 2024-02-27T06:30:23.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:23 smithi131 ceph-mon[66259]: pgmap v153: 97 pgs: 1 peering, 96 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.1 MiB/s wr, 373 op/s; 1.1 MiB/s, 23 objects/s recovering 2024-02-27T06:30:23.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:23 smithi131 ceph-mon[66259]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:23.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:23 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/1574479887' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:30:23.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:23 smithi150 ceph-mon[52803]: pgmap v153: 97 pgs: 1 peering, 96 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.1 MiB/s wr, 373 op/s; 1.1 MiB/s, 23 objects/s recovering 2024-02-27T06:30:23.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:23 smithi150 ceph-mon[52803]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:23.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:23 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/1574479887' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:30:24.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 238925 10.47 MB/sec execute 422 sec latency 39.851 ms 2024-02-27T06:30:25.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 239294 10.46 MB/sec execute 423 sec latency 62.013 ms 2024-02-27T06:30:25.691 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:25 smithi131 ceph-mon[66259]: pgmap v154: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.7 MiB/s wr, 351 op/s; 851 KiB/s, 17 objects/s recovering 2024-02-27T06:30:25.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:25 smithi150 ceph-mon[52803]: pgmap v154: 97 pgs: 97 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.7 MiB/s wr, 351 op/s; 851 KiB/s, 17 objects/s recovering 2024-02-27T06:30:26.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 239591 10.46 MB/sec execute 424 sec latency 39.264 ms 2024-02-27T06:30:26.966 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:26 smithi131 systemd[1]: Stopping Ceph osd.3 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:30:27.229 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:27 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[46245]: 2024-02-27T06:30:27.186+0000 7f67ec21c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:30:27.229 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:27 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[46245]: 2024-02-27T06:30:27.186+0000 7f67ec21c700 -1 osd.3 105 *** Got signal Terminated *** 2024-02-27T06:30:27.229 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:27 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[46245]: 2024-02-27T06:30:27.186+0000 7f67ec21c700 -1 osd.3 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:30:27.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 239886 10.46 MB/sec execute 425 sec latency 51.932 ms 2024-02-27T06:30:27.854 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:27 smithi131 ceph-mon[66259]: pgmap v155: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.5 MiB/s wr, 461 op/s; 756 KiB/s, 15 objects/s recovering 2024-02-27T06:30:27.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:27.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:30:27.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:27 smithi131 ceph-mon[66259]: osd.3 marked itself down and dead 2024-02-27T06:30:27.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:27 smithi150 ceph-mon[52803]: pgmap v155: 97 pgs: 97 active+clean; 335 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.5 MiB/s wr, 461 op/s; 756 KiB/s, 15 objects/s recovering 2024-02-27T06:30:27.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:27.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:30:27.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:27 smithi150 ceph-mon[52803]: osd.3 marked itself down and dead 2024-02-27T06:30:28.515 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:28 smithi131 podman[79842]: 2024-02-27 06:30:28.246665343 +0000 UTC m=+1.200972190 container died a85c70386ff64cfeefccd9924e79f9e6034aa3d0989ceeea47bf6ee45028a41b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , RELEASE=HEAD, ceph=True, com.redhat.component=centos-stream-container, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-27T06:30:28.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 239889 10.43 MB/sec execute 426 sec latency 989.115 ms 2024-02-27T06:30:28.765 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:28 smithi131 ceph-mon[66259]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:30:28.765 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:28 smithi131 ceph-mon[66259]: osdmap e106: 8 total, 7 up, 8 in 2024-02-27T06:30:28.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:28 smithi150 ceph-mon[52803]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:30:28.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:28 smithi150 ceph-mon[52803]: osdmap e106: 8 total, 7 up, 8 in 2024-02-27T06:30:29.026 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:28 smithi131 podman[79842]: 2024-02-27 06:30:28.808905514 +0000 UTC m=+1.763212354 container cleanup a85c70386ff64cfeefccd9924e79f9e6034aa3d0989ceeea47bf6ee45028a41b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., version=8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream) 2024-02-27T06:30:29.026 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:28 smithi131 bash[79842]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3 2024-02-27T06:30:29.279 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:29 smithi131 podman[79859]: 2024-02-27 06:30:29.161256316 +0000 UTC m=+0.918550267 container remove a85c70386ff64cfeefccd9924e79f9e6034aa3d0989ceeea47bf6ee45028a41b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, name=centos-stream, release=754, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, architecture=x86_64, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, ceph=True, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-27T06:30:29.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 240225 10.44 MB/sec execute 427 sec latency 1050.080 ms 2024-02-27T06:30:29.788 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:29 smithi131 ceph-mon[66259]: pgmap v157: 97 pgs: 16 stale+active+clean, 81 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 497 op/s; 106 KiB/s, 3 objects/s recovering 2024-02-27T06:30:29.788 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:29 smithi131 ceph-mon[66259]: osdmap e107: 8 total, 7 up, 8 in 2024-02-27T06:30:29.788 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:29 smithi131 podman[79940]: 2024-02-27 06:30:29.506048846 +0000 UTC m=+0.030733200 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:30:29.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:29 smithi150 ceph-mon[52803]: pgmap v157: 97 pgs: 16 stale+active+clean, 81 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.8 MiB/s wr, 497 op/s; 106 KiB/s, 3 objects/s recovering 2024-02-27T06:30:29.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:29 smithi150 ceph-mon[52803]: osdmap e107: 8 total, 7 up, 8 in 2024-02-27T06:30:30.039 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:29 smithi131 podman[79940]: 2024-02-27 06:30:29.780180718 +0000 UTC m=+0.304865050 container create 0716c60d50dcc371451f5fb03f8d801f60d176565247665f9a7468a2100cba16 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-27T06:30:30.550 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:30 smithi131 podman[79940]: 2024-02-27 06:30:30.302067955 +0000 UTC m=+0.826752311 container init 0716c60d50dcc371451f5fb03f8d801f60d176565247665f9a7468a2100cba16 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-27T06:30:30.551 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:30 smithi131 podman[79940]: 2024-02-27 06:30:30.309042359 +0000 UTC m=+0.833726694 container start 0716c60d50dcc371451f5fb03f8d801f60d176565247665f9a7468a2100cba16 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:30:30.551 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:30 smithi131 podman[79940]: 2024-02-27 06:30:30.50261511 +0000 UTC m=+1.027299450 container attach 0716c60d50dcc371451f5fb03f8d801f60d176565247665f9a7468a2100cba16 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-27T06:30:30.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 240663 10.44 MB/sec execute 428 sec latency 39.468 ms 2024-02-27T06:30:30.801 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:30 smithi131 podman[79940]: 2024-02-27 06:30:30.760368695 +0000 UTC m=+1.285053035 container died 0716c60d50dcc371451f5fb03f8d801f60d176565247665f9a7468a2100cba16 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:30:31.551 INFO:teuthology.orchestra.run.smithi131.stdout: 5 240997 10.43 MB/sec execute 429 sec latency 70.867 ms 2024-02-27T06:30:31.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:31 smithi131 ceph-mon[66259]: pgmap v159: 97 pgs: 4 active+undersized, 6 active+undersized+degraded, 12 stale+active+clean, 75 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 5.2 MiB/s wr, 526 op/s; 32/1662 objects degraded (1.925%) 2024-02-27T06:30:31.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:31 smithi131 ceph-mon[66259]: Health check failed: Degraded data redundancy: 32/1662 objects degraded (1.925%), 6 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:31 smithi150 ceph-mon[52803]: pgmap v159: 97 pgs: 4 active+undersized, 6 active+undersized+degraded, 12 stale+active+clean, 75 active+clean; 340 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 5.2 MiB/s wr, 526 op/s; 32/1662 objects degraded (1.925%) 2024-02-27T06:30:31.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:31 smithi150 ceph-mon[52803]: Health check failed: Degraded data redundancy: 32/1662 objects degraded (1.925%), 6 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:32.098 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:31 smithi131 podman[79965]: 2024-02-27 06:30:31.853880702 +0000 UTC m=+1.086222861 container remove 0716c60d50dcc371451f5fb03f8d801f60d176565247665f9a7468a2100cba16 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-02-27T06:30:32.098 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:31 smithi131 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.3.service: Deactivated successfully. 2024-02-27T06:30:32.098 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:31 smithi131 systemd[1]: Stopped Ceph osd.3 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:30:32.099 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:31 smithi131 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.3.service: Consumed 4min 31.166s CPU time. 2024-02-27T06:30:32.357 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:32 smithi131 systemd[1]: Starting Ceph osd.3 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:30:32.552 INFO:teuthology.orchestra.run.smithi131.stdout: 5 241190 10.41 MB/sec execute 430 sec latency 57.190 ms 2024-02-27T06:30:32.617 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:32 smithi131 podman[80067]: 2024-02-27 06:30:32.361530775 +0000 UTC m=+0.027153278 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:30:32.868 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:32 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:33.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:32 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:33.147 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:32 smithi131 podman[80067]: 2024-02-27 06:30:32.881793984 +0000 UTC m=+0.547416478 container create 8ddd641df2b97bf9c1573a847271136faeb51bd781dda81075d6791e286857b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:30:33.419 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 podman[80067]: 2024-02-27 06:30:33.263587848 +0000 UTC m=+0.929210336 container init 8ddd641df2b97bf9c1573a847271136faeb51bd781dda81075d6791e286857b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2) 2024-02-27T06:30:33.419 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 podman[80067]: 2024-02-27 06:30:33.268243658 +0000 UTC m=+0.933866147 container start 8ddd641df2b97bf9c1573a847271136faeb51bd781dda81075d6791e286857b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-02-27T06:30:33.419 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 podman[80067]: 2024-02-27 06:30:33.376716802 +0000 UTC m=+1.042339309 container attach 8ddd641df2b97bf9c1573a847271136faeb51bd781dda81075d6791e286857b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True) 2024-02-27T06:30:33.552 INFO:teuthology.orchestra.run.smithi131.stdout: 5 241387 10.40 MB/sec execute 431 sec latency 63.428 ms 2024-02-27T06:30:33.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:33 smithi131 ceph-mon[66259]: pgmap v160: 97 pgs: 7 active+undersized, 16 active+undersized+degraded, 8 stale+active+clean, 66 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.3 MiB/s wr, 475 op/s; 106/1698 objects degraded (6.243%) 2024-02-27T06:30:33.929 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate[80082]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-27T06:30:33.929 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate[80082]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-27T06:30:33.929 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 bash[80067]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-27T06:30:33.929 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 bash[80067]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-27T06:30:33.930 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 bash[80067]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-27T06:30:33.930 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 bash[80067]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-27T06:30:33.930 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 bash[80067]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-27T06:30:33.930 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 bash[80067]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-27T06:30:33.930 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 bash[80067]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-27T06:30:33.930 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate[80082]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-27T06:30:33.930 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate[80082]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-27T06:30:33.930 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate[80082]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-27T06:30:33.930 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate[80082]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-27T06:30:33.930 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate[80082]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-27T06:30:33.930 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:33 smithi131 podman[80067]: 2024-02-27 06:30:33.807153859 +0000 UTC m=+1.472776350 container died 8ddd641df2b97bf9c1573a847271136faeb51bd781dda81075d6791e286857b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5) 2024-02-27T06:30:34.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:33 smithi150 ceph-mon[52803]: pgmap v160: 97 pgs: 7 active+undersized, 16 active+undersized+degraded, 8 stale+active+clean, 66 active+clean; 338 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.3 MiB/s wr, 475 op/s; 106/1698 objects degraded (6.243%) 2024-02-27T06:30:34.552 INFO:teuthology.orchestra.run.smithi131.stdout: 5 241684 10.40 MB/sec execute 432 sec latency 86.136 ms 2024-02-27T06:30:34.982 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:34 smithi131 podman[80209]: 2024-02-27 06:30:34.952491942 +0000 UTC m=+1.139873103 container remove 8ddd641df2b97bf9c1573a847271136faeb51bd781dda81075d6791e286857b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0) 2024-02-27T06:30:35.242 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:35 smithi131 podman[80272]: 2024-02-27 06:30:35.104230093 +0000 UTC m=+0.024422997 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:30:35.552 INFO:teuthology.orchestra.run.smithi131.stdout: 5 241897 10.39 MB/sec execute 433 sec latency 88.066 ms 2024-02-27T06:30:35.769 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:35 smithi131 ceph-mon[66259]: pgmap v161: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.1 MiB/s wr, 389 op/s; 210/1683 objects degraded (12.478%) 2024-02-27T06:30:36.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:35 smithi150 ceph-mon[52803]: pgmap v161: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.1 MiB/s wr, 389 op/s; 210/1683 objects degraded (12.478%) 2024-02-27T06:30:36.281 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:36 smithi131 podman[80272]: 2024-02-27 06:30:36.04524666 +0000 UTC m=+0.965439591 container create 3d240baa2aedc2ebad6c8fbc11825c8e6aceae4fd792bc79347c88079be64405 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:30:36.534 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:36 smithi131 podman[80272]: 2024-02-27 06:30:36.450215738 +0000 UTC m=+1.370408629 container init 3d240baa2aedc2ebad6c8fbc11825c8e6aceae4fd792bc79347c88079be64405 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=) 2024-02-27T06:30:36.534 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:36 smithi131 podman[80272]: 2024-02-27 06:30:36.459256446 +0000 UTC m=+1.379449339 container start 3d240baa2aedc2ebad6c8fbc11825c8e6aceae4fd792bc79347c88079be64405 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-27T06:30:36.534 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:36 smithi131 ceph-osd[80290]: -- 172.21.15.131:0/3352916728 <== mon.0 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56020a5c7680 con 0x56020a590c00 2024-02-27T06:30:36.552 INFO:teuthology.orchestra.run.smithi131.stdout: 5 242217 10.39 MB/sec execute 434 sec latency 56.316 ms 2024-02-27T06:30:36.786 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:36 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 211/1677 objects degraded (12.582%), 27 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:36.786 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:36 smithi131 bash[80272]: 3d240baa2aedc2ebad6c8fbc11825c8e6aceae4fd792bc79347c88079be64405 2024-02-27T06:30:36.786 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:36 smithi131 systemd[1]: Started Ceph osd.3 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:30:37.109 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:36 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 211/1677 objects degraded (12.582%), 27 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:37.317 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:37 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[80286]: 2024-02-27T06:30:37.048+0000 7fc3949f27c0 -1 Falling back to public interface 2024-02-27T06:30:37.552 INFO:teuthology.orchestra.run.smithi131.stdout: 5 242531 10.39 MB/sec execute 435 sec latency 52.361 ms 2024-02-27T06:30:37.822 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:37 smithi131 ceph-mon[66259]: pgmap v162: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.8 MiB/s wr, 256 op/s; 211/1677 objects degraded (12.582%) 2024-02-27T06:30:37.822 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:37.822 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:37.822 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:30:38.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:37 smithi150 ceph-mon[52803]: pgmap v162: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 330 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.8 MiB/s wr, 256 op/s; 211/1677 objects degraded (12.582%) 2024-02-27T06:30:38.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:38.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:38.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:30:38.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 242969 10.38 MB/sec execute 436 sec latency 34.878 ms 2024-02-27T06:30:39.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 243334 10.38 MB/sec execute 437 sec latency 32.665 ms 2024-02-27T06:30:39.864 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:39 smithi131 ceph-mon[66259]: pgmap v163: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 332 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.7 MiB/s wr, 348 op/s; 202/1590 objects degraded (12.704%) 2024-02-27T06:30:40.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:39 smithi150 ceph-mon[52803]: pgmap v163: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 332 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.7 MiB/s wr, 348 op/s; 202/1590 objects degraded (12.704%) 2024-02-27T06:30:40.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 243814 10.39 MB/sec execute 438 sec latency 61.322 ms 2024-02-27T06:30:41.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 244363 10.40 MB/sec execute 439 sec latency 30.385 ms 2024-02-27T06:30:41.885 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:41 smithi131 ceph-mon[66259]: pgmap v164: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 397 op/s; 200/1608 objects degraded (12.438%) 2024-02-27T06:30:41.885 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:41.885 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:30:42.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:41 smithi150 ceph-mon[52803]: pgmap v164: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 397 op/s; 200/1608 objects degraded (12.438%) 2024-02-27T06:30:42.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:42.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:30:42.554 INFO:teuthology.orchestra.run.smithi131.stdout: 5 244685 10.39 MB/sec execute 440 sec latency 43.727 ms 2024-02-27T06:30:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:42 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 212/1671 objects degraded (12.687%), 27 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:43.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:42 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 212/1671 objects degraded (12.687%), 27 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:43.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 245020 10.39 MB/sec execute 441 sec latency 46.860 ms 2024-02-27T06:30:43.943 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:43 smithi131 ceph-mon[66259]: pgmap v165: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 371 op/s; 212/1671 objects degraded (12.687%) 2024-02-27T06:30:44.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:43 smithi150 ceph-mon[52803]: pgmap v165: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 331 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.9 MiB/s wr, 371 op/s; 212/1671 objects degraded (12.687%) 2024-02-27T06:30:44.194 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:44 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[80286]: 2024-02-27T06:30:44.124+0000 7fc3949f27c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-02-27T06:30:44.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 245354 10.39 MB/sec execute 442 sec latency 47.342 ms 2024-02-27T06:30:45.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 245680 10.38 MB/sec execute 443 sec latency 50.780 ms 2024-02-27T06:30:45.735 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:45 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[80286]: 2024-02-27T06:30:45.652+0000 7fc3949f27c0 -1 osd.3 105 log_to_monitors true 2024-02-27T06:30:46.004 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:45 smithi131 ceph-mon[66259]: pgmap v166: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.0 MiB/s wr, 494 op/s; 209/1731 objects degraded (12.074%) 2024-02-27T06:30:46.004 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:45 smithi131 ceph-mon[66259]: from='osd.3 [v2:172.21.15.131:6826/2190271523,v1:172.21.15.131:6827/2190271523]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T06:30:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:45 smithi150 ceph-mon[52803]: pgmap v166: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 335 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.0 MiB/s wr, 494 op/s; 209/1731 objects degraded (12.074%) 2024-02-27T06:30:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:45 smithi150 ceph-mon[52803]: from='osd.3 [v2:172.21.15.131:6826/2190271523,v1:172.21.15.131:6827/2190271523]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-27T06:30:46.512 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:30:46 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[80286]: 2024-02-27T06:30:46.296+0000 7fc381e42700 -1 osd.3 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:30:46.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 245938 10.38 MB/sec execute 444 sec latency 140.388 ms 2024-02-27T06:30:47.017 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:46 smithi131 ceph-mon[66259]: from='osd.3 [v2:172.21.15.131:6826/2190271523,v1:172.21.15.131:6827/2190271523]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-27T06:30:47.017 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:46 smithi131 ceph-mon[66259]: osdmap e108: 8 total, 7 up, 8 in 2024-02-27T06:30:47.017 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:46 smithi131 ceph-mon[66259]: from='osd.3 [v2:172.21.15.131:6826/2190271523,v1:172.21.15.131:6827/2190271523]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:30:47.017 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:46 smithi131 ceph-mon[66259]: from='osd.3 [v2:172.21.15.131:6826/2190271523,v1:172.21.15.131:6827/2190271523]' entity='osd.3' 2024-02-27T06:30:47.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:46 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:47.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:46 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:47.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:46 smithi150 ceph-mon[52803]: from='osd.3 [v2:172.21.15.131:6826/2190271523,v1:172.21.15.131:6827/2190271523]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-27T06:30:47.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:46 smithi150 ceph-mon[52803]: osdmap e108: 8 total, 7 up, 8 in 2024-02-27T06:30:47.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:46 smithi150 ceph-mon[52803]: from='osd.3 [v2:172.21.15.131:6826/2190271523,v1:172.21.15.131:6827/2190271523]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-02-27T06:30:47.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:46 smithi150 ceph-mon[52803]: from='osd.3 [v2:172.21.15.131:6826/2190271523,v1:172.21.15.131:6827/2190271523]' entity='osd.3' 2024-02-27T06:30:47.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:46 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:47.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:46 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:47.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 246127 10.37 MB/sec execute 445 sec latency 291.250 ms 2024-02-27T06:30:48.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:47 smithi131 ceph-mon[66259]: pgmap v168: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.8 MiB/s wr, 614 op/s; 201/1656 objects degraded (12.138%) 2024-02-27T06:30:48.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:47 smithi131 ceph-mon[66259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:30:48.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:47 smithi131 ceph-mon[66259]: osd.3 [v2:172.21.15.131:6826/2190271523,v1:172.21.15.131:6827/2190271523] boot 2024-02-27T06:30:48.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:47 smithi131 ceph-mon[66259]: osdmap e109: 8 total, 8 up, 8 in 2024-02-27T06:30:48.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:30:48.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:47 smithi150 ceph-mon[52803]: pgmap v168: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 5.8 MiB/s wr, 614 op/s; 201/1656 objects degraded (12.138%) 2024-02-27T06:30:48.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:47 smithi150 ceph-mon[52803]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:30:48.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:47 smithi150 ceph-mon[52803]: osd.3 [v2:172.21.15.131:6826/2190271523,v1:172.21.15.131:6827/2190271523] boot 2024-02-27T06:30:48.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:47 smithi150 ceph-mon[52803]: osdmap e109: 8 total, 8 up, 8 in 2024-02-27T06:30:48.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-27T06:30:48.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 246127 10.34 MB/sec execute 446 sec latency 1291.329 ms 2024-02-27T06:30:49.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:49.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:49.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:48 smithi131 ceph-mon[66259]: osdmap e110: 8 total, 8 up, 8 in 2024-02-27T06:30:49.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:49.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:49.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:48 smithi150 ceph-mon[52803]: osdmap e110: 8 total, 8 up, 8 in 2024-02-27T06:30:49.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 246127 10.32 MB/sec execute 447 sec latency 2291.388 ms 2024-02-27T06:30:50.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:49 smithi131 ceph-mon[66259]: pgmap v171: 97 pgs: 10 peering, 9 active+undersized, 22 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 6.3 MiB/s wr, 635 op/s; 149/1641 objects degraded (9.080%) 2024-02-27T06:30:50.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:49 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 149/1641 objects degraded (9.080%), 22 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:50.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:49 smithi131 ceph-mon[66259]: osdmap e111: 8 total, 8 up, 8 in 2024-02-27T06:30:50.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:49 smithi150 ceph-mon[52803]: pgmap v171: 97 pgs: 10 peering, 9 active+undersized, 22 active+undersized+degraded, 56 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 6.3 MiB/s wr, 635 op/s; 149/1641 objects degraded (9.080%) 2024-02-27T06:30:50.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:49 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 149/1641 objects degraded (9.080%), 22 pgs degraded (PG_DEGRADED) 2024-02-27T06:30:50.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:49 smithi150 ceph-mon[52803]: osdmap e111: 8 total, 8 up, 8 in 2024-02-27T06:30:50.554 INFO:teuthology.orchestra.run.smithi131.stdout: 5 246289 10.31 MB/sec execute 448 sec latency 2570.758 ms 2024-02-27T06:30:51.509 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:51 smithi131 ceph-mon[66259]: osdmap e112: 8 total, 8 up, 8 in 2024-02-27T06:30:51.509 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:51 smithi131 ceph-mon[66259]: pgmap v174: 97 pgs: 10 peering, 7 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 17 active+undersized+degraded, 60 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.1 MiB/s wr, 130 op/s; 148/1611 objects degraded (9.187%); 1.0 MiB/s, 5 objects/s recovering 2024-02-27T06:30:51.509 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:51 smithi131 ceph-mon[66259]: osdmap e113: 8 total, 8 up, 8 in 2024-02-27T06:30:51.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 246599 10.29 MB/sec execute 449 sec latency 177.730 ms 2024-02-27T06:30:51.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:51 smithi150 ceph-mon[52803]: osdmap e112: 8 total, 8 up, 8 in 2024-02-27T06:30:51.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:51 smithi150 ceph-mon[52803]: pgmap v174: 97 pgs: 10 peering, 7 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 17 active+undersized+degraded, 60 active+clean; 332 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.1 MiB/s wr, 130 op/s; 148/1611 objects degraded (9.187%); 1.0 MiB/s, 5 objects/s recovering 2024-02-27T06:30:51.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:51 smithi150 ceph-mon[52803]: osdmap e113: 8 total, 8 up, 8 in 2024-02-27T06:30:52.529 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:52 smithi131 ceph-mon[66259]: osdmap e114: 8 total, 8 up, 8 in 2024-02-27T06:30:52.553 INFO:teuthology.orchestra.run.smithi131.stdout: 5 246808 10.29 MB/sec execute 450 sec latency 107.744 ms 2024-02-27T06:30:52.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:52 smithi150 ceph-mon[52803]: osdmap e114: 8 total, 8 up, 8 in 2024-02-27T06:30:53.518 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: pgmap v177: 97 pgs: 11 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+remapped, 81 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 1.6 MiB/s wr, 198 op/s; 1/1614 objects degraded (0.062%); 3.5 MiB/s, 54 objects/s recovering 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: osdmap e115: 8 total, 8 up, 8 in 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T06:30:53.519 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:53 smithi131 ceph-mon[66259]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-02-27T06:30:53.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 247042 10.28 MB/sec execute 451 sec latency 60.771 ms 2024-02-27T06:30:53.798 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:30:53.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: pgmap v177: 97 pgs: 11 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+remapped, 81 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 1.6 MiB/s wr, 198 op/s; 1/1614 objects degraded (0.062%); 3.5 MiB/s, 54 objects/s recovering 2024-02-27T06:30:53.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: osdmap e115: 8 total, 8 up, 8 in 2024-02-27T06:30:53.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:53.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:53.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:30:53.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:30:53.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:53.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:53.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:30:53.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:53.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:53.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:53.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T06:30:53.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T06:30:53.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:53 smithi150 ceph-mon[52803]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-02-27T06:30:54.536 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:54 smithi131 ceph-mon[66259]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:54.536 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:54 smithi131 ceph-mon[66259]: from='client.34338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:54.555 INFO:teuthology.orchestra.run.smithi131.stdout: 5 247326 10.28 MB/sec execute 452 sec latency 50.073 ms 2024-02-27T06:30:54.743 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:30:54.743 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (7m) 8s ago 15m 22.0M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:30:54.743 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (3m) 8s ago 15m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:30:54.743 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (3m) 2m ago 15m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:30:54.743 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (6m) 8s ago 15m 51.9M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (10m) 8s ago 10m 39.1M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (10m) 2m ago 10m 14.2M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (5m) 8s ago 17m 508M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (5m) 2m ago 15m 449M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (4m) 8s ago 17m 49.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (4m) 2m ago 15m 36.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (9m) 8s ago 9m 277M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (9m) 2m ago 9m 81.4M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (6m) 8s ago 15m 17.9M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (6m) 2m ago 15m 17.2M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (2m) 8s ago 14m 334M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (100s) 8s ago 13m 285M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (52s) 8s ago 13m 199M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:30:54.744 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (19s) 8s ago 13m 14.9M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 3d240baa2aed 2024-02-27T06:30:54.745 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (12m) 2m ago 12m 557M 2704M 17.2.0 e1d6a67b021e 024932f8763b 2024-02-27T06:30:54.745 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (12m) 2m ago 12m 436M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:30:54.745 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (11m) 2m ago 11m 347M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:30:54.745 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (11m) 2m ago 11m 542M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:30:54.745 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (5m) 8s ago 14m 68.0M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:30:54.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:54 smithi150 ceph-mon[52803]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:54.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:54 smithi150 ceph-mon[52803]: from='client.34338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:55.218 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:30:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:30:55.218 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 4 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:30:55.219 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:30:55.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 247602 10.27 MB/sec execute 453 sec latency 74.930 ms 2024-02-27T06:30:55.572 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:55 smithi131 ceph-mon[66259]: pgmap v179: 97 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+remapped, 91 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.5 MiB/s wr, 290 op/s; 1/1617 objects degraded (0.062%); 2.4 MiB/s, 62 objects/s recovering 2024-02-27T06:30:55.572 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:55 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 1/1617 objects degraded (0.062%), 1 pg degraded (PG_DEGRADED) 2024-02-27T06:30:55.572 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:55 smithi131 ceph-mon[66259]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:55.572 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:55 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/1471306717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:55.771 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:30:55.771 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:30:55.771 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:30:55.771 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:30:55.771 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:30:55.771 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:30:55.771 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:30:55.771 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:30:55.771 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:30:55.771 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "10/23 daemons upgraded", 2024-02-27T06:30:55.771 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T06:30:55.771 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:30:55.772 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:30:55.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:55 smithi150 ceph-mon[52803]: pgmap v179: 97 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+remapped, 91 active+clean; 334 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.5 MiB/s wr, 290 op/s; 1/1617 objects degraded (0.062%); 2.4 MiB/s, 62 objects/s recovering 2024-02-27T06:30:55.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:55 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 1/1617 objects degraded (0.062%), 1 pg degraded (PG_DEGRADED) 2024-02-27T06:30:55.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:55 smithi150 ceph-mon[52803]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:55.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:55 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/1471306717' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:30:56.265 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_WARN Degraded data redundancy: 1/1617 objects degraded (0.062%), 1 pg degraded 2024-02-27T06:30:56.265 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 1/1617 objects degraded (0.062%), 1 pg degraded 2024-02-27T06:30:56.265 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.d is active+recovery_wait+undersized+degraded+remapped, acting [1,4] 2024-02-27T06:30:56.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 247926 10.27 MB/sec execute 454 sec latency 81.394 ms 2024-02-27T06:30:56.598 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:56 smithi131 ceph-mon[66259]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:56.598 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:56 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/568655498' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:30:56.598 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:56 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:56.598 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:56 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:30:56.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:56 smithi150 ceph-mon[52803]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:30:56.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:56 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/568655498' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:30:56.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:56 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:30:56.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:56 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:30:57.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 248186 10.26 MB/sec execute 455 sec latency 46.443 ms 2024-02-27T06:30:57.633 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:57 smithi131 ceph-mon[66259]: pgmap v180: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 3.2 MiB/s wr, 283 op/s; 2.3 MiB/s, 45 objects/s recovering 2024-02-27T06:30:57.633 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:57 smithi131 ceph-mon[66259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1617 objects degraded (0.062%), 1 pg degraded) 2024-02-27T06:30:57.633 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:57 smithi131 ceph-mon[66259]: Cluster is now healthy 2024-02-27T06:30:57.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:57 smithi150 ceph-mon[52803]: pgmap v180: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 3.2 MiB/s wr, 283 op/s; 2.3 MiB/s, 45 objects/s recovering 2024-02-27T06:30:57.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:57 smithi150 ceph-mon[52803]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1617 objects degraded (0.062%), 1 pg degraded) 2024-02-27T06:30:57.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:57 smithi150 ceph-mon[52803]: Cluster is now healthy 2024-02-27T06:30:58.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 248468 10.25 MB/sec execute 456 sec latency 46.178 ms 2024-02-27T06:30:59.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 248811 10.25 MB/sec execute 457 sec latency 53.861 ms 2024-02-27T06:30:59.687 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:30:59 smithi131 ceph-mon[66259]: pgmap v181: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.0 MiB/s wr, 436 op/s; 1.7 MiB/s, 34 objects/s recovering 2024-02-27T06:30:59.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:30:59 smithi150 ceph-mon[52803]: pgmap v181: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.0 MiB/s wr, 436 op/s; 1.7 MiB/s, 34 objects/s recovering 2024-02-27T06:31:00.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 249125 10.25 MB/sec execute 458 sec latency 55.305 ms 2024-02-27T06:31:01.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 249394 10.25 MB/sec execute 459 sec latency 55.294 ms 2024-02-27T06:31:01.759 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:01 smithi131 ceph-mon[66259]: pgmap v182: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.2 MiB/s wr, 316 op/s; 436 KiB/s, 8 objects/s recovering 2024-02-27T06:31:01.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:01 smithi150 ceph-mon[52803]: pgmap v182: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.2 MiB/s wr, 316 op/s; 436 KiB/s, 8 objects/s recovering 2024-02-27T06:31:02.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 249665 10.24 MB/sec execute 460 sec latency 53.858 ms 2024-02-27T06:31:02.838 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:02 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:03.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:02 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:03.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 250011 10.23 MB/sec execute 461 sec latency 84.995 ms 2024-02-27T06:31:03.863 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:03 smithi131 ceph-mon[66259]: pgmap v183: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.4 MiB/s wr, 307 op/s; 392 KiB/s, 7 objects/s recovering 2024-02-27T06:31:04.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:03 smithi150 ceph-mon[52803]: pgmap v183: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.4 MiB/s wr, 307 op/s; 392 KiB/s, 7 objects/s recovering 2024-02-27T06:31:04.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 250281 10.22 MB/sec execute 462 sec latency 39.819 ms 2024-02-27T06:31:05.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 250566 10.22 MB/sec execute 463 sec latency 47.692 ms 2024-02-27T06:31:05.643 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:05 smithi131 ceph-mon[66259]: pgmap v184: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.1 MiB/s wr, 413 op/s; 327 KiB/s, 6 objects/s recovering 2024-02-27T06:31:06.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:05 smithi150 ceph-mon[52803]: pgmap v184: 97 pgs: 97 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.1 MiB/s wr, 413 op/s; 327 KiB/s, 6 objects/s recovering 2024-02-27T06:31:06.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 250859 10.22 MB/sec execute 464 sec latency 43.045 ms 2024-02-27T06:31:07.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 251151 10.21 MB/sec execute 465 sec latency 51.722 ms 2024-02-27T06:31:07.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:07 smithi131 ceph-mon[66259]: pgmap v185: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.4 MiB/s wr, 457 op/s; 326 KiB/s, 0 objects/s recovering 2024-02-27T06:31:08.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:07 smithi150 ceph-mon[52803]: pgmap v185: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.4 MiB/s wr, 457 op/s; 326 KiB/s, 0 objects/s recovering 2024-02-27T06:31:08.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 251451 10.21 MB/sec execute 466 sec latency 61.294 ms 2024-02-27T06:31:08.698 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:08 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T06:31:08.699 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:08 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T06:31:08.699 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:08 smithi131 ceph-mon[66259]: Upgrade: osd.4 is safe to restart 2024-02-27T06:31:08.778 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:08 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T06:31:08.778 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:08 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-27T06:31:08.778 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:08 smithi150 ceph-mon[52803]: Upgrade: osd.4 is safe to restart 2024-02-27T06:31:09.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 251671 10.20 MB/sec execute 467 sec latency 50.736 ms 2024-02-27T06:31:09.736 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:09 smithi131 ceph-mon[66259]: pgmap v186: 97 pgs: 97 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.5 MiB/s wr, 470 op/s 2024-02-27T06:31:09.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:09 smithi150 ceph-mon[52803]: pgmap v186: 97 pgs: 97 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.5 MiB/s wr, 470 op/s 2024-02-27T06:31:10.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 251892 10.19 MB/sec execute 468 sec latency 67.904 ms 2024-02-27T06:31:10.754 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:10 smithi131 ceph-mon[66259]: Upgrade: Updating osd.4 2024-02-27T06:31:10.754 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:10 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:10.755 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:10 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T06:31:10.755 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:10 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:31:10.755 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:10 smithi131 ceph-mon[66259]: Deploying daemon osd.4 on smithi150 2024-02-27T06:31:10.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:10 smithi150 ceph-mon[52803]: Upgrade: Updating osd.4 2024-02-27T06:31:10.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:10 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:10.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:10 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-27T06:31:10.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:10 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:31:10.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:10 smithi150 ceph-mon[52803]: Deploying daemon osd.4 on smithi150 2024-02-27T06:31:11.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 252175 10.18 MB/sec execute 469 sec latency 56.580 ms 2024-02-27T06:31:11.788 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:11 smithi131 ceph-mon[66259]: pgmap v187: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.7 MiB/s wr, 330 op/s 2024-02-27T06:31:11.788 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:11.789 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:31:11.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:11 smithi150 ceph-mon[52803]: pgmap v187: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.7 MiB/s wr, 330 op/s 2024-02-27T06:31:11.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:11.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:31:12.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 252583 10.18 MB/sec execute 470 sec latency 37.487 ms 2024-02-27T06:31:13.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 252930 10.19 MB/sec execute 471 sec latency 34.622 ms 2024-02-27T06:31:13.610 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:13 smithi150 systemd[1]: Stopping Ceph osd.4 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:31:13.830 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:13 smithi131 ceph-mon[66259]: pgmap v188: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.4 MiB/s wr, 452 op/s 2024-02-27T06:31:14.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:13 smithi150 ceph-mon[52803]: pgmap v188: 97 pgs: 97 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.4 MiB/s wr, 452 op/s 2024-02-27T06:31:14.110 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:13 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[30957]: 2024-02-27T06:31:13.706+0000 7f4b39a3b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:31:14.110 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:13 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[30957]: 2024-02-27T06:31:13.706+0000 7f4b39a3b700 -1 osd.4 115 *** Got signal Terminated *** 2024-02-27T06:31:14.110 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:13 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[30957]: 2024-02-27T06:31:13.706+0000 7f4b39a3b700 -1 osd.4 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:31:14.556 INFO:teuthology.orchestra.run.smithi131.stdout: 5 253285 10.19 MB/sec execute 472 sec latency 41.396 ms 2024-02-27T06:31:14.837 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:14 smithi131 ceph-mon[66259]: osd.4 marked itself down and dead 2024-02-27T06:31:15.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:14 smithi150 ceph-mon[52803]: osd.4 marked itself down and dead 2024-02-27T06:31:15.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 253420 10.17 MB/sec execute 473 sec latency 819.855 ms 2024-02-27T06:31:15.608 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:15 smithi150 podman[57259]: 2024-02-27 06:31:15.323600609 +0000 UTC m=+1.844398208 container died 024932f8763b343b85f23ab191da99a44effa9587dfb739114e7b3e39aedebf9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-type=git, GIT_BRANCH=HEAD, name=centos-stream, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-02-27T06:31:15.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:15 smithi150 ceph-mon[52803]: pgmap v189: 97 pgs: 97 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.8 MiB/s wr, 509 op/s 2024-02-27T06:31:15.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:15 smithi150 ceph-mon[52803]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:31:15.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:15 smithi150 ceph-mon[52803]: osdmap e116: 8 total, 7 up, 8 in 2024-02-27T06:31:15.861 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:15 smithi150 podman[57259]: 2024-02-27 06:31:15.690677221 +0000 UTC m=+2.211474820 container cleanup 024932f8763b343b85f23ab191da99a44effa9587dfb739114e7b3e39aedebf9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_BRANCH=HEAD, architecture=x86_64, ceph=True, release=754, version=8, RELEASE=HEAD, distribution-scope=public, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-27T06:31:15.861 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:15 smithi150 bash[57259]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4 2024-02-27T06:31:15.948 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:15 smithi131 ceph-mon[66259]: pgmap v189: 97 pgs: 97 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.8 MiB/s wr, 509 op/s 2024-02-27T06:31:15.948 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:15 smithi131 ceph-mon[66259]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:31:15.948 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:15 smithi131 ceph-mon[66259]: osdmap e116: 8 total, 7 up, 8 in 2024-02-27T06:31:16.358 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:15 smithi150 podman[57275]: 2024-02-27 06:31:15.975676024 +0000 UTC m=+0.648248771 container remove 024932f8763b343b85f23ab191da99a44effa9587dfb739114e7b3e39aedebf9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4, release=754, name=centos-stream, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-27T06:31:16.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 253704 10.16 MB/sec execute 474 sec latency 1042.476 ms 2024-02-27T06:31:16.610 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:16 smithi150 podman[57364]: 2024-02-27 06:31:16.25299528 +0000 UTC m=+0.019520695 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:31:16.610 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:16 smithi150 podman[57364]: 2024-02-27 06:31:16.404506868 +0000 UTC m=+0.171032290 container create b200e3b4d9023baccc1e3242ff9461bab60f53b630ed99caadac01d7d6a47de1 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-27T06:31:16.969 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:16 smithi131 ceph-mon[66259]: osdmap e117: 8 total, 7 up, 8 in 2024-02-27T06:31:16.974 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:16 smithi150 ceph-mon[52803]: osdmap e117: 8 total, 7 up, 8 in 2024-02-27T06:31:16.974 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:16 smithi150 podman[57364]: 2024-02-27 06:31:16.646332335 +0000 UTC m=+0.412857749 container init b200e3b4d9023baccc1e3242ff9461bab60f53b630ed99caadac01d7d6a47de1 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2) 2024-02-27T06:31:16.974 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:16 smithi150 podman[57364]: 2024-02-27 06:31:16.650204019 +0000 UTC m=+0.416729439 container start b200e3b4d9023baccc1e3242ff9461bab60f53b630ed99caadac01d7d6a47de1 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-27T06:31:16.974 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:16 smithi150 podman[57364]: 2024-02-27 06:31:16.721487778 +0000 UTC m=+0.488013209 container attach b200e3b4d9023baccc1e3242ff9461bab60f53b630ed99caadac01d7d6a47de1 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:31:17.277 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:16 smithi150 podman[57364]: 2024-02-27 06:31:16.965843758 +0000 UTC m=+0.732369189 container died b200e3b4d9023baccc1e3242ff9461bab60f53b630ed99caadac01d7d6a47de1 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:31:17.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 253961 10.15 MB/sec execute 475 sec latency 68.624 ms 2024-02-27T06:31:17.742 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:17 smithi131 ceph-mon[66259]: pgmap v192: 97 pgs: 5 active+undersized, 8 active+undersized+degraded, 9 peering, 5 stale+active+clean, 70 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.2 MiB/s wr, 403 op/s; 71/1692 objects degraded (4.196%) 2024-02-27T06:31:17.742 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:17 smithi131 ceph-mon[66259]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-27T06:31:17.743 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:17 smithi131 ceph-mon[66259]: Health check failed: Degraded data redundancy: 71/1692 objects degraded (4.196%), 8 pgs degraded (PG_DEGRADED) 2024-02-27T06:31:17.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:17 smithi150 ceph-mon[52803]: pgmap v192: 97 pgs: 5 active+undersized, 8 active+undersized+degraded, 9 peering, 5 stale+active+clean, 70 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.2 MiB/s wr, 403 op/s; 71/1692 objects degraded (4.196%) 2024-02-27T06:31:17.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:17 smithi150 ceph-mon[52803]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-27T06:31:17.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:17 smithi150 ceph-mon[52803]: Health check failed: Degraded data redundancy: 71/1692 objects degraded (4.196%), 8 pgs degraded (PG_DEGRADED) 2024-02-27T06:31:17.861 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:17 smithi150 podman[57390]: 2024-02-27 06:31:17.570032481 +0000 UTC m=+0.597986295 container remove b200e3b4d9023baccc1e3242ff9461bab60f53b630ed99caadac01d7d6a47de1 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:31:17.861 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:17 smithi150 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-27T06:31:17.861 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:17 smithi150 systemd[1]: Stopped Ceph osd.4 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:31:17.861 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:17 smithi150 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.4.service: Consumed 4min 55.767s CPU time. 2024-02-27T06:31:17.861 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:17 smithi150 systemd[1]: Starting Ceph osd.4 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:31:18.350 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 podman[57489]: 2024-02-27 06:31:17.93890249 +0000 UTC m=+0.019838538 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:31:18.350 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 podman[57489]: 2024-02-27 06:31:18.080076772 +0000 UTC m=+0.161012818 container create 41b45ba629b9e574838446ec13794e34737e6360eb924d4cf59b312de57d3ae4 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:31:18.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 254241 10.15 MB/sec execute 476 sec latency 54.067 ms 2024-02-27T06:31:18.610 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 podman[57489]: 2024-02-27 06:31:18.341469453 +0000 UTC m=+0.422405508 container init 41b45ba629b9e574838446ec13794e34737e6360eb924d4cf59b312de57d3ae4 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:31:18.610 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 podman[57489]: 2024-02-27 06:31:18.344954205 +0000 UTC m=+0.425890261 container start 41b45ba629b9e574838446ec13794e34737e6360eb924d4cf59b312de57d3ae4 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS) 2024-02-27T06:31:18.610 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 podman[57489]: 2024-02-27 06:31:18.44128621 +0000 UTC m=+0.522222274 container attach 41b45ba629b9e574838446ec13794e34737e6360eb924d4cf59b312de57d3ae4 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, ceph=True) 2024-02-27T06:31:19.111 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate[57504]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-27T06:31:19.111 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate[57504]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-27T06:31:19.111 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 bash[57489]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-27T06:31:19.111 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 bash[57489]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-27T06:31:19.111 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 bash[57489]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-27T06:31:19.111 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 bash[57489]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-27T06:31:19.111 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 bash[57489]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-27T06:31:19.111 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 bash[57489]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-27T06:31:19.111 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 bash[57489]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-27T06:31:19.111 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate[57504]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-27T06:31:19.112 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate[57504]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-27T06:31:19.112 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate[57504]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-27T06:31:19.112 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate[57504]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-27T06:31:19.112 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate[57504]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-27T06:31:19.112 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:18 smithi150 podman[57489]: 2024-02-27 06:31:18.868305989 +0000 UTC m=+0.949242045 container died 41b45ba629b9e574838446ec13794e34737e6360eb924d4cf59b312de57d3ae4 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:31:19.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 254543 10.15 MB/sec execute 477 sec latency 56.345 ms 2024-02-27T06:31:19.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:19 smithi131 ceph-mon[66259]: pgmap v193: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 9 peering, 56 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.7 MiB/s wr, 507 op/s; 220/1644 objects degraded (13.382%) 2024-02-27T06:31:19.922 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:19 smithi150 ceph-mon[52803]: pgmap v193: 97 pgs: 9 active+undersized, 23 active+undersized+degraded, 9 peering, 56 active+clean; 338 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.7 MiB/s wr, 507 op/s; 220/1644 objects degraded (13.382%) 2024-02-27T06:31:19.923 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:19 smithi150 podman[57626]: 2024-02-27 06:31:19.65477189 +0000 UTC m=+0.777650559 container remove 41b45ba629b9e574838446ec13794e34737e6360eb924d4cf59b312de57d3ae4 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0) 2024-02-27T06:31:19.923 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:19 smithi150 podman[57688]: 2024-02-27 06:31:19.779166602 +0000 UTC m=+0.022378452 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:31:20.226 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:19 smithi150 podman[57688]: 2024-02-27 06:31:19.9142874 +0000 UTC m=+0.157499236 container create 7dc53922c3097b5c188f676604f2a17945c4db584d15d1f9fd7356aed6c5c34e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:31:20.226 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:20 smithi150 podman[57688]: 2024-02-27 06:31:20.125265327 +0000 UTC m=+0.368477156 container init 7dc53922c3097b5c188f676604f2a17945c4db584d15d1f9fd7356aed6c5c34e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:31:20.226 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:20 smithi150 podman[57688]: 2024-02-27 06:31:20.128330342 +0000 UTC m=+0.371542164 container start 7dc53922c3097b5c188f676604f2a17945c4db584d15d1f9fd7356aed6c5c34e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-27T06:31:20.226 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:20 smithi150 ceph-osd[57706]: -- 172.21.15.150:0/4041809128 <== mon.1 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558840835680 con 0x5588407fec00 2024-02-27T06:31:20.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 254873 10.15 MB/sec execute 478 sec latency 75.896 ms 2024-02-27T06:31:20.610 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:20 smithi150 bash[57688]: 7dc53922c3097b5c188f676604f2a17945c4db584d15d1f9fd7356aed6c5c34e 2024-02-27T06:31:20.610 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:20 smithi150 systemd[1]: Started Ceph osd.4 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:31:21.110 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:20 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:31:20.676+0000 7f51f3b2b7c0 -1 Falling back to public interface 2024-02-27T06:31:21.374 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:21 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:21.374 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:21 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:21.374 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:21 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:31:21.374 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:21 smithi131 ceph-mon[66259]: pgmap v194: 97 pgs: 12 active+undersized, 27 active+undersized+degraded, 2 peering, 56 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.3 MiB/s wr, 443 op/s; 249/1626 objects degraded (15.314%) 2024-02-27T06:31:21.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 255116 10.14 MB/sec execute 479 sec latency 88.336 ms 2024-02-27T06:31:21.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:21 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:21.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:21 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:21.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:21 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:31:21.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:21 smithi150 ceph-mon[52803]: pgmap v194: 97 pgs: 12 active+undersized, 27 active+undersized+degraded, 2 peering, 56 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.3 MiB/s wr, 443 op/s; 249/1626 objects degraded (15.314%) 2024-02-27T06:31:22.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 255361 10.13 MB/sec execute 480 sec latency 46.726 ms 2024-02-27T06:31:22.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:22 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 264/1650 objects degraded (16.000%), 28 pgs degraded (PG_DEGRADED) 2024-02-27T06:31:22.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:22 smithi131 ceph-mon[66259]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-27T06:31:22.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:22 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 264/1650 objects degraded (16.000%), 28 pgs degraded (PG_DEGRADED) 2024-02-27T06:31:22.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:22 smithi150 ceph-mon[52803]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-27T06:31:23.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 255560 10.12 MB/sec execute 481 sec latency 53.270 ms 2024-02-27T06:31:23.697 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:23 smithi131 ceph-mon[66259]: pgmap v195: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.1 MiB/s wr, 427 op/s; 264/1650 objects degraded (16.000%) 2024-02-27T06:31:23.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:23 smithi150 ceph-mon[52803]: pgmap v195: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.1 MiB/s wr, 427 op/s; 264/1650 objects degraded (16.000%) 2024-02-27T06:31:24.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 255817 10.11 MB/sec execute 482 sec latency 71.936 ms 2024-02-27T06:31:25.299 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:25.299 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:25.299 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:25 smithi131 ceph-mon[66259]: pgmap v196: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.9 MiB/s wr, 382 op/s; 262/1635 objects degraded (16.024%) 2024-02-27T06:31:25.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 256087 10.11 MB/sec execute 483 sec latency 56.679 ms 2024-02-27T06:31:25.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:25.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:25.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:25 smithi150 ceph-mon[52803]: pgmap v196: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.9 MiB/s wr, 382 op/s; 262/1635 objects degraded (16.024%) 2024-02-27T06:31:26.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 256307 10.10 MB/sec execute 484 sec latency 53.874 ms 2024-02-27T06:31:26.745 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:31:26.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:26.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:26.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:26.847 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:31:27.078 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:27.078 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:27.079 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:27.079 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:31:27.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 256515 10.09 MB/sec execute 485 sec latency 52.204 ms 2024-02-27T06:31:27.715 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:27 smithi150 ceph-mon[52803]: pgmap v197: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.1 MiB/s wr, 401 op/s; 272/1659 objects degraded (16.395%) 2024-02-27T06:31:27.715 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:27 smithi150 ceph-mon[52803]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:31:27.766 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:31:27.766 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (7m) 41s ago 16m 22.0M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:31:27.766 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (3m) 41s ago 16m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:31:27.766 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (3m) 3s ago 15m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:31:27.766 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (7m) 41s ago 16m 51.9M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:31:27.766 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (10m) 41s ago 10m 39.1M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:31:27.766 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (10m) 3s ago 10m 15.0M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:31:27.766 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (6m) 41s ago 17m 508M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:31:27.766 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (5m) 3s ago 15m 451M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (5m) 41s ago 17m 49.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (4m) 3s ago 15m 41.0M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (10m) 41s ago 10m 277M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (10m) 3s ago 10m 83.9M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (7m) 41s ago 16m 17.9M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (6m) 3s ago 15m 17.6M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (3m) 41s ago 15m 334M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (2m) 41s ago 14m 285M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (85s) 41s ago 14m 199M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (52s) 41s ago 13m 14.9M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 3d240baa2aed 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (7s) 3s ago 13m 15.1M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 7dc53922c309 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (12m) 3s ago 12m 601M 2704M 17.2.0 e1d6a67b021e 96af4aebab24 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (12m) 3s ago 12m 535M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (12m) 3s ago 12m 574M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:31:27.767 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (5m) 41s ago 15m 68.0M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:31:27.890 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:27 smithi131 ceph-mon[66259]: pgmap v197: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 343 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.1 MiB/s wr, 401 op/s; 272/1659 objects degraded (16.395%) 2024-02-27T06:31:27.891 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:27 smithi131 ceph-mon[66259]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:31:27.983 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:27 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:31:27.704+0000 7f51f3b2b7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-02-27T06:31:28.392 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:31:28.392 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 4 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-27T06:31:28.393 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T06:31:28.394 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:31:28.394 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:31:28.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 256902 10.09 MB/sec execute 486 sec latency 70.976 ms 2024-02-27T06:31:28.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:28 smithi131 ceph-mon[66259]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:31:28.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:28 smithi131 ceph-mon[66259]: from='client.34366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:31:28.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:28 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/75293288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:31:28.935 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:31:28.935 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:31:28.935 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:31:28.935 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:31:28.935 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:31:28.936 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:31:28.936 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:31:28.936 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:31:28.936 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:31:28.936 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "11/23 daemons upgraded", 2024-02-27T06:31:28.936 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T06:31:28.936 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:31:28.936 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:31:29.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:28 smithi150 ceph-mon[52803]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:31:29.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:28 smithi150 ceph-mon[52803]: from='client.34366 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:31:29.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:28 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/75293288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:31:29.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 257300 10.08 MB/sec execute 487 sec latency 55.476 ms 2024-02-27T06:31:29.581 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 279/1677 objects degraded (16.637%), 28 pgs degraded 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: osd.4 (root=default,host=smithi150) is down 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 279/1677 objects degraded (16.637%), 28 pgs degraded 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.5 is active+undersized+degraded, acting [7,0] 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.6 is active+undersized+degraded, acting [1,6] 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.b is active+undersized+degraded, acting [7,5] 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.f is active+undersized+degraded, acting [0,2] 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.11 is active+undersized+degraded, acting [6,1] 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.13 is active+undersized+degraded, acting [0,3] 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.18 is active+undersized+degraded, acting [5,7] 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.1a is active+undersized+degraded, acting [6,7] 2024-02-27T06:31:29.582 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.1f is active+undersized+degraded, acting [0,7] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.e is active+undersized+degraded, acting [7,1] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-02-27T06:31:29.583 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-02-27T06:31:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-02-27T06:31:29.584 INFO:teuthology.orchestra.run.smithi131.stdout: pg 4.5 is active+undersized+degraded, acting [6,7] 2024-02-27T06:31:29.610 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:29 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:31:29.192+0000 7f51f3b2b7c0 -1 osd.4 115 log_to_monitors true 2024-02-27T06:31:29.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:29 smithi131 ceph-mon[66259]: pgmap v198: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.1 MiB/s wr, 447 op/s; 279/1677 objects degraded (16.637%) 2024-02-27T06:31:29.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:29 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 279/1677 objects degraded (16.637%), 28 pgs degraded (PG_DEGRADED) 2024-02-27T06:31:29.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:29 smithi131 ceph-mon[66259]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:31:29.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:29 smithi131 ceph-mon[66259]: from='osd.4 [v2:172.21.15.150:6800/2461983337,v1:172.21.15.150:6801/2461983337]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T06:31:29.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:29 smithi131 ceph-mon[66259]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T06:31:29.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:29 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/3212718098' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:31:30.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:29 smithi150 ceph-mon[52803]: pgmap v198: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 347 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 4.1 MiB/s wr, 447 op/s; 279/1677 objects degraded (16.637%) 2024-02-27T06:31:30.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:29 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 279/1677 objects degraded (16.637%), 28 pgs degraded (PG_DEGRADED) 2024-02-27T06:31:30.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:29 smithi150 ceph-mon[52803]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:31:30.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:29 smithi150 ceph-mon[52803]: from='osd.4 [v2:172.21.15.150:6800/2461983337,v1:172.21.15.150:6801/2461983337]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T06:31:30.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:29 smithi150 ceph-mon[52803]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-27T06:31:30.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:29 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/3212718098' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:31:30.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 257582 10.08 MB/sec execute 488 sec latency 104.237 ms 2024-02-27T06:31:30.610 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:31:30.192+0000 7f51e0f7b700 -1 osd.4 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:31:30.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-27T06:31:30.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='osd.4 [v2:172.21.15.150:6800/2461983337,v1:172.21.15.150:6801/2461983337]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:31:30.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: osdmap e118: 8 total, 7 up, 8 in 2024-02-27T06:31:30.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:31:30.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:30.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:30.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:31:30.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:31:30.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:30.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:30.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='osd.4 ' entity='osd.4' 2024-02-27T06:31:30.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:31:30.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:31:30.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:31:30.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:31:30.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T06:31:30.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:30 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T06:31:31.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-27T06:31:31.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='osd.4 [v2:172.21.15.150:6800/2461983337,v1:172.21.15.150:6801/2461983337]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: osdmap e118: 8 total, 7 up, 8 in 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='osd.4 ' entity='osd.4' 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:31:31.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T06:31:31.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:30 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T06:31:31.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 257906 10.09 MB/sec execute 489 sec latency 366.469 ms 2024-02-27T06:31:32.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:31 smithi131 ceph-mon[66259]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-02-27T06:31:32.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:31 smithi131 ceph-mon[66259]: pgmap v200: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 347 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.8 MiB/s wr, 354 op/s; 279/1680 objects degraded (16.607%) 2024-02-27T06:31:32.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:31 smithi131 ceph-mon[66259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:31:32.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:31 smithi131 ceph-mon[66259]: osd.4 [v2:172.21.15.150:6800/2461983337,v1:172.21.15.150:6801/2461983337] boot 2024-02-27T06:31:32.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:31 smithi131 ceph-mon[66259]: osdmap e119: 8 total, 8 up, 8 in 2024-02-27T06:31:32.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:31:32.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:31 smithi150 ceph-mon[52803]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-02-27T06:31:32.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:31 smithi150 ceph-mon[52803]: pgmap v200: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 347 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.8 MiB/s wr, 354 op/s; 279/1680 objects degraded (16.607%) 2024-02-27T06:31:32.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:31 smithi150 ceph-mon[52803]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:31:32.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:31 smithi150 ceph-mon[52803]: osd.4 [v2:172.21.15.150:6800/2461983337,v1:172.21.15.150:6801/2461983337] boot 2024-02-27T06:31:32.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:31 smithi150 ceph-mon[52803]: osdmap e119: 8 total, 8 up, 8 in 2024-02-27T06:31:32.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-27T06:31:32.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 257906 10.07 MB/sec execute 490 sec latency 1366.514 ms 2024-02-27T06:31:33.467 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:33 smithi131 ceph-mon[66259]: osdmap e120: 8 total, 8 up, 8 in 2024-02-27T06:31:33.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 258013 10.05 MB/sec execute 491 sec latency 2082.123 ms 2024-02-27T06:31:33.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:33 smithi150 ceph-mon[52803]: osdmap e120: 8 total, 8 up, 8 in 2024-02-27T06:31:34.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:34 smithi131 ceph-mon[66259]: pgmap v203: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 349 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 523 op/s; 282/1656 objects degraded (17.029%) 2024-02-27T06:31:34.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:34 smithi131 ceph-mon[66259]: osdmap e121: 8 total, 8 up, 8 in 2024-02-27T06:31:34.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 258373 10.04 MB/sec execute 492 sec latency 356.786 ms 2024-02-27T06:31:34.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:34 smithi150 ceph-mon[52803]: pgmap v203: 97 pgs: 13 active+undersized, 28 active+undersized+degraded, 56 active+clean; 349 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 523 op/s; 282/1656 objects degraded (17.029%) 2024-02-27T06:31:34.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:34 smithi150 ceph-mon[52803]: osdmap e121: 8 total, 8 up, 8 in 2024-02-27T06:31:35.481 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:35 smithi131 ceph-mon[66259]: osdmap e122: 8 total, 8 up, 8 in 2024-02-27T06:31:35.557 INFO:teuthology.orchestra.run.smithi131.stdout: 5 258468 10.03 MB/sec execute 493 sec latency 1116.207 ms 2024-02-27T06:31:35.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:35 smithi150 ceph-mon[52803]: osdmap e122: 8 total, 8 up, 8 in 2024-02-27T06:31:36.268 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:36 smithi131 ceph-mon[66259]: pgmap v206: 97 pgs: 1 remapped+peering, 1 active+recovering+degraded, 6 active+recovery_wait+degraded, 13 active+undersized, 14 active+undersized+degraded, 62 active+clean; 351 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.4 MiB/s wr, 496 op/s; 241/1659 objects degraded (14.527%); 0 B/s, 0 keys/s, 3 objects/s recovering 2024-02-27T06:31:36.268 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:36 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 241/1659 objects degraded (14.527%), 21 pgs degraded (PG_DEGRADED) 2024-02-27T06:31:36.269 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:36 smithi131 ceph-mon[66259]: osdmap e123: 8 total, 8 up, 8 in 2024-02-27T06:31:36.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 258714 10.02 MB/sec execute 494 sec latency 946.029 ms 2024-02-27T06:31:36.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:36 smithi150 ceph-mon[52803]: pgmap v206: 97 pgs: 1 remapped+peering, 1 active+recovering+degraded, 6 active+recovery_wait+degraded, 13 active+undersized, 14 active+undersized+degraded, 62 active+clean; 351 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.4 MiB/s wr, 496 op/s; 241/1659 objects degraded (14.527%); 0 B/s, 0 keys/s, 3 objects/s recovering 2024-02-27T06:31:36.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:36 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 241/1659 objects degraded (14.527%), 21 pgs degraded (PG_DEGRADED) 2024-02-27T06:31:36.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:36 smithi150 ceph-mon[52803]: osdmap e123: 8 total, 8 up, 8 in 2024-02-27T06:31:37.279 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:37 smithi131 ceph-mon[66259]: osdmap e124: 8 total, 8 up, 8 in 2024-02-27T06:31:37.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 258935 10.02 MB/sec execute 495 sec latency 1016.395 ms 2024-02-27T06:31:37.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:37 smithi150 ceph-mon[52803]: osdmap e124: 8 total, 8 up, 8 in 2024-02-27T06:31:38.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:38 smithi131 ceph-mon[66259]: pgmap v209: 97 pgs: 4 peering, 1 remapped+peering, 1 active+recovering+degraded, 6 active+recovery_wait+degraded, 85 active+clean; 345 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 2.9 MiB/s wr, 251 op/s; 96/1635 objects degraded (5.872%); 1.8 MiB/s, 57 keys/s, 40 objects/s recovering 2024-02-27T06:31:38.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:38 smithi131 ceph-mon[66259]: osdmap e125: 8 total, 8 up, 8 in 2024-02-27T06:31:38.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 259245 10.01 MB/sec execute 496 sec latency 44.323 ms 2024-02-27T06:31:38.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:38 smithi150 ceph-mon[52803]: pgmap v209: 97 pgs: 4 peering, 1 remapped+peering, 1 active+recovering+degraded, 6 active+recovery_wait+degraded, 85 active+clean; 345 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 2.9 MiB/s wr, 251 op/s; 96/1635 objects degraded (5.872%); 1.8 MiB/s, 57 keys/s, 40 objects/s recovering 2024-02-27T06:31:38.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:38 smithi150 ceph-mon[52803]: osdmap e125: 8 total, 8 up, 8 in 2024-02-27T06:31:39.352 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:39 smithi131 ceph-mon[66259]: pgmap v211: 97 pgs: 4 peering, 93 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 2.2 MiB/s wr, 247 op/s; 1.7 MiB/s, 53 keys/s, 65 objects/s recovering 2024-02-27T06:31:39.352 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:39 smithi131 ceph-mon[66259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 96/1635 objects degraded (5.872%), 7 pgs degraded) 2024-02-27T06:31:39.352 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:39 smithi131 ceph-mon[66259]: Cluster is now healthy 2024-02-27T06:31:39.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 259554 10.01 MB/sec execute 497 sec latency 43.073 ms 2024-02-27T06:31:39.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:39 smithi150 ceph-mon[52803]: pgmap v211: 97 pgs: 4 peering, 93 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 2.2 MiB/s wr, 247 op/s; 1.7 MiB/s, 53 keys/s, 65 objects/s recovering 2024-02-27T06:31:39.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:39 smithi150 ceph-mon[52803]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 96/1635 objects degraded (5.872%), 7 pgs degraded) 2024-02-27T06:31:39.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:39 smithi150 ceph-mon[52803]: Cluster is now healthy 2024-02-27T06:31:40.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 259769 10.00 MB/sec execute 498 sec latency 85.515 ms 2024-02-27T06:31:41.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 260071 10.01 MB/sec execute 499 sec latency 52.660 ms 2024-02-27T06:31:41.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:41 smithi131 ceph-mon[66259]: pgmap v212: 97 pgs: 2 peering, 95 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.0 MiB/s wr, 283 op/s; 1.4 MiB/s, 37 keys/s, 49 objects/s recovering 2024-02-27T06:31:41.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:41.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:31:41.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:41 smithi150 ceph-mon[52803]: pgmap v212: 97 pgs: 2 peering, 95 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.0 MiB/s wr, 283 op/s; 1.4 MiB/s, 37 keys/s, 49 objects/s recovering 2024-02-27T06:31:41.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:41.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:31:42.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 260275 9.99 MB/sec execute 500 sec latency 107.687 ms 2024-02-27T06:31:43.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 260518 9.98 MB/sec execute 501 sec latency 55.967 ms 2024-02-27T06:31:43.740 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:43 smithi131 ceph-mon[66259]: pgmap v213: 97 pgs: 97 active+clean; 344 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.7 MiB/s wr, 189 op/s; 877 KiB/s, 30 objects/s recovering 2024-02-27T06:31:43.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:43 smithi150 ceph-mon[52803]: pgmap v213: 97 pgs: 97 active+clean; 344 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.7 MiB/s wr, 189 op/s; 877 KiB/s, 30 objects/s recovering 2024-02-27T06:31:44.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 260739 9.98 MB/sec execute 502 sec latency 61.988 ms 2024-02-27T06:31:45.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 260967 9.97 MB/sec execute 503 sec latency 72.418 ms 2024-02-27T06:31:45.833 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:45 smithi131 ceph-mon[66259]: pgmap v214: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.4 MiB/s wr, 335 op/s; 135 KiB/s, 17 objects/s recovering 2024-02-27T06:31:45.834 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:45 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T06:31:45.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:45 smithi150 ceph-mon[52803]: pgmap v214: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.4 MiB/s wr, 335 op/s; 135 KiB/s, 17 objects/s recovering 2024-02-27T06:31:45.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:45 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T06:31:46.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 261228 9.96 MB/sec execute 504 sec latency 48.108 ms 2024-02-27T06:31:46.605 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:46 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T06:31:46.605 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:46 smithi131 ceph-mon[66259]: Upgrade: osd.5 is safe to restart 2024-02-27T06:31:46.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:46 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-27T06:31:46.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:46 smithi150 ceph-mon[52803]: Upgrade: osd.5 is safe to restart 2024-02-27T06:31:47.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 261480 9.95 MB/sec execute 505 sec latency 60.999 ms 2024-02-27T06:31:48.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:48 smithi131 ceph-mon[66259]: pgmap v215: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.6 MiB/s wr, 316 op/s; 112 KiB/s, 14 objects/s recovering 2024-02-27T06:31:48.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:48 smithi131 ceph-mon[66259]: Upgrade: Updating osd.5 2024-02-27T06:31:48.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:48.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T06:31:48.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:31:48.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:48 smithi131 ceph-mon[66259]: Deploying daemon osd.5 on smithi150 2024-02-27T06:31:48.520 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:48 smithi150 ceph-mon[52803]: pgmap v215: 97 pgs: 97 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.6 MiB/s wr, 316 op/s; 112 KiB/s, 14 objects/s recovering 2024-02-27T06:31:48.520 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:48 smithi150 ceph-mon[52803]: Upgrade: Updating osd.5 2024-02-27T06:31:48.520 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:48.520 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-27T06:31:48.520 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:31:48.520 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:48 smithi150 ceph-mon[52803]: Deploying daemon osd.5 on smithi150 2024-02-27T06:31:48.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 261741 9.94 MB/sec execute 506 sec latency 60.449 ms 2024-02-27T06:31:49.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 262026 9.94 MB/sec execute 507 sec latency 57.080 ms 2024-02-27T06:31:50.450 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:50 smithi131 ceph-mon[66259]: pgmap v216: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.9 MiB/s wr, 412 op/s; 99 KiB/s, 12 objects/s recovering 2024-02-27T06:31:50.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 262294 9.93 MB/sec execute 508 sec latency 59.228 ms 2024-02-27T06:31:50.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:50 smithi150 ceph-mon[52803]: pgmap v216: 97 pgs: 97 active+clean; 344 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.9 MiB/s wr, 412 op/s; 99 KiB/s, 12 objects/s recovering 2024-02-27T06:31:51.333 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:51 smithi150 systemd[1]: Stopping Ceph osd.5 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:31:51.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 262578 9.93 MB/sec execute 509 sec latency 92.949 ms 2024-02-27T06:31:51.610 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:51 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[33943]: 2024-02-27T06:31:51.322+0000 7fd4d828f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:31:51.610 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:51 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[33943]: 2024-02-27T06:31:51.322+0000 7fd4d828f700 -1 osd.5 125 *** Got signal Terminated *** 2024-02-27T06:31:51.610 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:51 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[33943]: 2024-02-27T06:31:51.322+0000 7fd4d828f700 -1 osd.5 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:31:52.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:52 smithi131 ceph-mon[66259]: pgmap v217: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.8 MiB/s wr, 371 op/s; 93 KiB/s, 1 objects/s recovering 2024-02-27T06:31:52.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:52 smithi131 ceph-mon[66259]: osd.5 marked itself down and dead 2024-02-27T06:31:52.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:52 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:52.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 262768 9.92 MB/sec execute 510 sec latency 348.294 ms 2024-02-27T06:31:52.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:52 smithi150 ceph-mon[52803]: pgmap v217: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.8 MiB/s wr, 371 op/s; 93 KiB/s, 1 objects/s recovering 2024-02-27T06:31:52.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:52 smithi150 ceph-mon[52803]: osd.5 marked itself down and dead 2024-02-27T06:31:52.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:52 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:53.191 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:52 smithi150 podman[59520]: 2024-02-27 06:31:52.870923306 +0000 UTC m=+1.666966448 container died 96af4aebab249920c12346d2b6731ab7f9ae61e7bff1389cb73ec571193663ba (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, release=754, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, vcs-type=git, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.buildah.version=1.19.8, ceph=True, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-27T06:31:53.477 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:53 smithi131 ceph-mon[66259]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:31:53.477 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:53 smithi131 ceph-mon[66259]: osdmap e126: 8 total, 7 up, 8 in 2024-02-27T06:31:53.542 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:53 smithi150 ceph-mon[52803]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:31:53.542 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:53 smithi150 ceph-mon[52803]: osdmap e126: 8 total, 7 up, 8 in 2024-02-27T06:31:53.543 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:53 smithi150 podman[59520]: 2024-02-27 06:31:53.273442718 +0000 UTC m=+2.069485865 container cleanup 96af4aebab249920c12346d2b6731ab7f9ae61e7bff1389cb73ec571193663ba (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, name=centos-stream, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-27T06:31:53.543 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:53 smithi150 bash[59520]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5 2024-02-27T06:31:53.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 262826 9.90 MB/sec execute 511 sec latency 1102.315 ms 2024-02-27T06:31:53.860 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:53 smithi150 podman[59537]: 2024-02-27 06:31:53.533592396 +0000 UTC m=+0.654534187 container remove 96af4aebab249920c12346d2b6731ab7f9ae61e7bff1389cb73ec571193663ba (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5, maintainer=Guillaume Abrioux , release=754, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, vcs-type=git, io.openshift.expose-services=, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, distribution-scope=public, name=centos-stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, architecture=x86_64) 2024-02-27T06:31:54.230 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:53 smithi150 podman[59627]: 2024-02-27 06:31:53.803125308 +0000 UTC m=+0.019957819 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:31:54.230 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:53 smithi150 podman[59627]: 2024-02-27 06:31:53.917992904 +0000 UTC m=+0.134825408 container create 1c050b0cbd9c3f42b973be718b6afef71a9deddc4af5e0c175b3a7d49e792c9e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, ceph=True) 2024-02-27T06:31:54.230 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:54 smithi150 podman[59627]: 2024-02-27 06:31:54.129511351 +0000 UTC m=+0.346343861 container init 1c050b0cbd9c3f42b973be718b6afef71a9deddc4af5e0c175b3a7d49e792c9e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-02-27T06:31:54.230 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:54 smithi150 podman[59627]: 2024-02-27 06:31:54.133184747 +0000 UTC m=+0.350017259 container start 1c050b0cbd9c3f42b973be718b6afef71a9deddc4af5e0c175b3a7d49e792c9e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:31:54.230 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:54 smithi150 podman[59627]: 2024-02-27 06:31:54.220980096 +0000 UTC m=+0.437812613 container attach 1c050b0cbd9c3f42b973be718b6afef71a9deddc4af5e0c175b3a7d49e792c9e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-deactivate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-02-27T06:31:54.242 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:54 smithi131 ceph-mon[66259]: pgmap v219: 97 pgs: 12 stale+active+clean, 85 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.7 MiB/s wr, 387 op/s 2024-02-27T06:31:54.243 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:54 smithi131 ceph-mon[66259]: osdmap e127: 8 total, 7 up, 8 in 2024-02-27T06:31:54.482 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:54 smithi150 ceph-mon[52803]: pgmap v219: 97 pgs: 12 stale+active+clean, 85 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.7 MiB/s wr, 387 op/s 2024-02-27T06:31:54.482 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:54 smithi150 ceph-mon[52803]: osdmap e127: 8 total, 7 up, 8 in 2024-02-27T06:31:54.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 263111 9.90 MB/sec execute 512 sec latency 49.912 ms 2024-02-27T06:31:54.800 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:54 smithi150 podman[59627]: 2024-02-27 06:31:54.473934002 +0000 UTC m=+0.690766511 container died 1c050b0cbd9c3f42b973be718b6afef71a9deddc4af5e0c175b3a7d49e792c9e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:31:55.527 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:55 smithi131 ceph-mon[66259]: pgmap v221: 97 pgs: 11 peering, 86 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 3.5 MiB/s wr, 428 op/s 2024-02-27T06:31:55.527 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:55 smithi131 ceph-mon[66259]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-27T06:31:55.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 263506 9.91 MB/sec execute 513 sec latency 35.473 ms 2024-02-27T06:31:55.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:55 smithi150 ceph-mon[52803]: pgmap v221: 97 pgs: 11 peering, 86 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 3.5 MiB/s wr, 428 op/s 2024-02-27T06:31:55.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:55 smithi150 ceph-mon[52803]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-27T06:31:55.611 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:55 smithi150 podman[59653]: 2024-02-27 06:31:55.460265581 +0000 UTC m=+0.979847441 container remove 1c050b0cbd9c3f42b973be718b6afef71a9deddc4af5e0c175b3a7d49e792c9e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:31:55.611 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:55 smithi150 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.5.service: Deactivated successfully. 2024-02-27T06:31:55.611 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:55 smithi150 systemd[1]: Stopped Ceph osd.5 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:31:55.611 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:55 smithi150 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.5.service: Consumed 5min 24.317s CPU time. 2024-02-27T06:31:56.110 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:55 smithi150 systemd[1]: Starting Ceph osd.5 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:31:56.110 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 podman[59756]: 2024-02-27 06:31:55.926260438 +0000 UTC m=+0.019136529 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:31:56.482 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 podman[59756]: 2024-02-27 06:31:56.203142897 +0000 UTC m=+0.296018974 container create e1acfb70bbaa495fd20fb80b7d4bc12160c413ea3aa6db36245fa1ed37a20be2 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:31:56.482 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 podman[59756]: 2024-02-27 06:31:56.473075243 +0000 UTC m=+0.565951330 container init e1acfb70bbaa495fd20fb80b7d4bc12160c413ea3aa6db36245fa1ed37a20be2 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-02-27T06:31:56.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 263890 9.91 MB/sec execute 514 sec latency 40.345 ms 2024-02-27T06:31:56.860 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 podman[59756]: 2024-02-27 06:31:56.476756275 +0000 UTC m=+0.569632363 container start e1acfb70bbaa495fd20fb80b7d4bc12160c413ea3aa6db36245fa1ed37a20be2 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0) 2024-02-27T06:31:56.860 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 podman[59756]: 2024-02-27 06:31:56.589783915 +0000 UTC m=+0.682660004 container attach e1acfb70bbaa495fd20fb80b7d4bc12160c413ea3aa6db36245fa1ed37a20be2 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate[59771]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate[59771]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 bash[59756]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 bash[59756]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 bash[59756]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 bash[59756]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 bash[59756]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 bash[59756]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 bash[59756]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate[59771]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate[59771]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate[59771]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-27T06:31:57.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate[59771]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-27T06:31:57.362 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate[59771]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-27T06:31:57.362 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:57 smithi150 podman[59756]: 2024-02-27 06:31:57.015703434 +0000 UTC m=+1.108579513 container died e1acfb70bbaa495fd20fb80b7d4bc12160c413ea3aa6db36245fa1ed37a20be2 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:31:57.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 264250 9.91 MB/sec execute 515 sec latency 37.639 ms 2024-02-27T06:31:57.572 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:57 smithi131 ceph-mon[66259]: pgmap v222: 97 pgs: 4 active+undersized, 15 active+undersized+degraded, 11 peering, 67 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.7 MiB/s wr, 337 op/s; 111/1653 objects degraded (6.715%) 2024-02-27T06:31:57.573 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:57.573 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:31:57.573 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:57 smithi131 ceph-mon[66259]: Health check failed: Degraded data redundancy: 111/1653 objects degraded (6.715%), 15 pgs degraded (PG_DEGRADED) 2024-02-27T06:31:57.787 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:57 smithi150 ceph-mon[52803]: pgmap v222: 97 pgs: 4 active+undersized, 15 active+undersized+degraded, 11 peering, 67 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.7 MiB/s wr, 337 op/s; 111/1653 objects degraded (6.715%) 2024-02-27T06:31:57.787 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:57.787 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:31:57.787 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:57 smithi150 ceph-mon[52803]: Health check failed: Degraded data redundancy: 111/1653 objects degraded (6.715%), 15 pgs degraded (PG_DEGRADED) 2024-02-27T06:31:58.110 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:57 smithi150 podman[59887]: 2024-02-27 06:31:57.77866303 +0000 UTC m=+0.756772193 container remove e1acfb70bbaa495fd20fb80b7d4bc12160c413ea3aa6db36245fa1ed37a20be2 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-02-27T06:31:58.110 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:57 smithi150 podman[59952]: 2024-02-27 06:31:57.903655594 +0000 UTC m=+0.022783170 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:31:58.111 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:58 smithi150 podman[59952]: 2024-02-27 06:31:58.02133758 +0000 UTC m=+0.140465137 container create 6865822553f8db19745b1f70e1c16804d0d5ebfe57a58e311a64b5b73264b59a (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0) 2024-02-27T06:31:58.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 264537 9.90 MB/sec execute 516 sec latency 67.656 ms 2024-02-27T06:31:58.611 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:58 smithi150 podman[59952]: 2024-02-27 06:31:58.282652788 +0000 UTC m=+0.401780357 container init 6865822553f8db19745b1f70e1c16804d0d5ebfe57a58e311a64b5b73264b59a (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-27T06:31:58.611 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:58 smithi150 podman[59952]: 2024-02-27 06:31:58.285883273 +0000 UTC m=+0.405010834 container start 6865822553f8db19745b1f70e1c16804d0d5ebfe57a58e311a64b5b73264b59a (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-02-27T06:31:58.611 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:58 smithi150 ceph-osd[59971]: -- 172.21.15.150:0/614049898 <== mon.1 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55955bb1d680 con 0x55955bae6c00 2024-02-27T06:31:58.611 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:58 smithi150 bash[59952]: 6865822553f8db19745b1f70e1c16804d0d5ebfe57a58e311a64b5b73264b59a 2024-02-27T06:31:58.611 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:58 smithi150 systemd[1]: Started Ceph osd.5 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:31:59.110 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:31:58 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:31:58.827+0000 7f00b60ce7c0 -1 Falling back to public interface 2024-02-27T06:31:59.558 INFO:teuthology.orchestra.run.smithi131.stdout: 5 264791 9.89 MB/sec execute 517 sec latency 76.170 ms 2024-02-27T06:31:59.613 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:59 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:59.614 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:59 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:59.614 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:59 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:31:59.614 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:31:59 smithi131 ceph-mon[66259]: pgmap v223: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.8 MiB/s wr, 395 op/s; 173/1647 objects degraded (10.504%) 2024-02-27T06:31:59.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:59 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:59.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:59 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:31:59.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:59 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:31:59.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:31:59 smithi150 ceph-mon[52803]: pgmap v223: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.8 MiB/s wr, 395 op/s; 173/1647 objects degraded (10.504%) 2024-02-27T06:32:00.110 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:32:00.494 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:00 smithi150 ceph-mon[52803]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-02-27T06:32:00.494 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:00 smithi150 ceph-mon[52803]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:00.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 265038 9.89 MB/sec execute 518 sec latency 65.069 ms 2024-02-27T06:32:00.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:00 smithi131 ceph-mon[66259]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-02-27T06:32:00.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:00 smithi131 ceph-mon[66259]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:01.133 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:32:01.133 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (8m) 74s ago 17m 22.0M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:32:01.133 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (4m) 74s ago 16m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:32:01.133 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (4m) 36s ago 16m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:32:01.133 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (7m) 74s ago 16m 51.9M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:32:01.133 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (11m) 74s ago 11m 39.1M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:32:01.133 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (11m) 36s ago 11m 15.0M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:32:01.133 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (7m) 74s ago 18m 508M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:32:01.133 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (6m) 36s ago 16m 451M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (5m) 74s ago 18m 49.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (5m) 36s ago 16m 41.0M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (10m) 74s ago 10m 277M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (10m) 36s ago 10m 83.9M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (7m) 74s ago 16m 17.9M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (7m) 36s ago 16m 17.6M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (3m) 74s ago 15m 334M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (2m) 74s ago 15m 285M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (118s) 74s ago 14m 199M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (86s) 74s ago 14m 14.9M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 3d240baa2aed 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (41s) 36s ago 13m 15.1M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 7dc53922c309 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 starting - - - 2704M 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (13m) 36s ago 13m 535M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (12m) 36s ago 12m 574M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:32:01.134 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (6m) 74s ago 15m 68.0M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:32:01.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 265308 9.88 MB/sec execute 519 sec latency 110.921 ms 2024-02-27T06:32:01.700 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:01 smithi131 ceph-mon[66259]: pgmap v224: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 347 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 441 op/s; 176/1689 objects degraded (10.420%) 2024-02-27T06:32:01.700 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:01 smithi131 ceph-mon[66259]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:01.700 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:01 smithi131 ceph-mon[66259]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:01.731 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:32:01.731 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:32:01.731 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:32:01.731 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:32:01.731 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:32:01.731 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:32:01.731 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:32:01.732 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:32:01.732 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-27T06:32:01.732 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 5 2024-02-27T06:32:01.732 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:32:01.732 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:32:01.732 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:32:01.732 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:32:01.732 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:32:01.732 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-27T06:32:01.732 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 9 2024-02-27T06:32:01.732 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:32:01.732 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:32:01.748 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:01 smithi150 ceph-mon[52803]: pgmap v224: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 347 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 441 op/s; 176/1689 objects degraded (10.420%) 2024-02-27T06:32:01.748 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:01 smithi150 ceph-mon[52803]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:01.748 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:01 smithi150 ceph-mon[52803]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:02.207 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:32:02.207 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:32:02.208 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:32:02.208 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:32:02.208 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:32:02.208 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:32:02.208 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:32:02.208 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:32:02.208 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:32:02.208 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "11/23 daemons upgraded", 2024-02-27T06:32:02.208 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T06:32:02.208 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:32:02.208 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:32:02.474 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:02 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/2568208424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:02.474 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:02 smithi131 ceph-mon[66259]: from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:02.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 265572 9.88 MB/sec execute 520 sec latency 69.126 ms 2024-02-27T06:32:02.719 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:02 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/2568208424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:02.719 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:02 smithi150 ceph-mon[52803]: from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:02.730 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 176/1689 objects degraded (10.420%), 23 pgs degraded 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: osd.5 (root=default,host=smithi150) is down 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 176/1689 objects degraded (10.420%), 23 pgs degraded 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.3 is active+undersized+degraded, acting [2,7] 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.b is active+undersized+degraded, acting [7,4] 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.18 is active+undersized+degraded, acting [4,7] 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.1c is active+undersized+degraded, acting [7,2] 2024-02-27T06:32:02.731 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.1e is active+undersized+degraded, acting [3,0] 2024-02-27T06:32:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-02-27T06:32:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-02-27T06:32:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-02-27T06:32:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-02-27T06:32:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-02-27T06:32:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-02-27T06:32:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-02-27T06:32:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-02-27T06:32:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-02-27T06:32:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-02-27T06:32:02.732 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-02-27T06:32:03.495 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:03 smithi131 ceph-mon[66259]: pgmap v225: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.3 MiB/s wr, 474 op/s; 171/1644 objects degraded (10.401%) 2024-02-27T06:32:03.495 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:03 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/1687735520' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:32:03.495 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:03 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:03.495 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:03 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:03.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 265852 9.87 MB/sec execute 521 sec latency 49.175 ms 2024-02-27T06:32:03.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:03 smithi150 ceph-mon[52803]: pgmap v225: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.3 MiB/s wr, 474 op/s; 171/1644 objects degraded (10.401%) 2024-02-27T06:32:03.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:03 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/1687735520' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:32:03.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:03 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:03.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:03 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:04.534 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:04 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 171/1644 objects degraded (10.401%), 23 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:04.534 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:04 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:04.534 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:04 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:04.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 266201 9.87 MB/sec execute 522 sec latency 59.015 ms 2024-02-27T06:32:04.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:04 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 171/1644 objects degraded (10.401%), 23 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:04.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:04 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:04.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:04 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:05.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 266394 9.86 MB/sec execute 523 sec latency 66.642 ms 2024-02-27T06:32:05.561 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:05 smithi131 ceph-mon[66259]: pgmap v226: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 321 op/s; 174/1629 objects degraded (10.681%) 2024-02-27T06:32:05.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:05 smithi150 ceph-mon[52803]: pgmap v226: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 344 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 321 op/s; 174/1629 objects degraded (10.681%) 2024-02-27T06:32:06.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 266664 9.86 MB/sec execute 524 sec latency 63.774 ms 2024-02-27T06:32:07.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 266919 9.85 MB/sec execute 525 sec latency 63.558 ms 2024-02-27T06:32:07.566 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:32:07 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:32:07.290+0000 7f00b60ce7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-02-27T06:32:07.620 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:07 smithi131 ceph-mon[66259]: pgmap v227: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 345 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 389 op/s; 187/1641 objects degraded (11.395%) 2024-02-27T06:32:07.828 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:07 smithi150 ceph-mon[52803]: pgmap v227: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 345 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 389 op/s; 187/1641 objects degraded (11.395%) 2024-02-27T06:32:08.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 267214 9.86 MB/sec execute 526 sec latency 55.419 ms 2024-02-27T06:32:09.110 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:32:08 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:32:08.702+0000 7f00b60ce7c0 -1 osd.5 125 log_to_monitors true 2024-02-27T06:32:09.429 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:09.429 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:09.429 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:32:09.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:32:09.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:09.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:09.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:32:09.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:09.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:09.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:09.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T06:32:09.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='osd.5 [v2:172.21.15.150:6808/2623515039,v1:172.21.15.150:6809/2623515039]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T06:32:09.430 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:09 smithi131 ceph-mon[66259]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T06:32:09.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 267454 9.85 MB/sec execute 527 sec latency 146.192 ms 2024-02-27T06:32:09.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:09.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:09.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:32:09.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:32:09.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:09.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:09.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:32:09.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:09.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:09.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:09.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T06:32:09.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='osd.5 [v2:172.21.15.150:6808/2623515039,v1:172.21.15.150:6809/2623515039]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T06:32:09.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-mon[52803]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-27T06:32:10.110 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:32:09 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:32:09.735+0000 7f00a351e700 -1 osd.5 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:32:10.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:10 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T06:32:10.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:10 smithi131 ceph-mon[66259]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-27T06:32:10.446 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:10 smithi131 ceph-mon[66259]: pgmap v228: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.1 MiB/s wr, 426 op/s; 186/1644 objects degraded (11.314%) 2024-02-27T06:32:10.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:10 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 186/1644 objects degraded (11.314%), 23 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:10.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:10 smithi131 ceph-mon[66259]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-27T06:32:10.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:10 smithi131 ceph-mon[66259]: osdmap e128: 8 total, 7 up, 8 in 2024-02-27T06:32:10.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:10 smithi131 ceph-mon[66259]: from='osd.5 [v2:172.21.15.150:6808/2623515039,v1:172.21.15.150:6809/2623515039]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:32:10.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:10 smithi131 ceph-mon[66259]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:32:10.447 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:10 smithi131 ceph-mon[66259]: from='osd.5 ' entity='osd.5' 2024-02-27T06:32:10.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 267702 9.84 MB/sec execute 528 sec latency 151.621 ms 2024-02-27T06:32:10.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:10 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T06:32:10.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:10 smithi150 ceph-mon[52803]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-27T06:32:10.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:10 smithi150 ceph-mon[52803]: pgmap v228: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.1 MiB/s wr, 426 op/s; 186/1644 objects degraded (11.314%) 2024-02-27T06:32:10.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:10 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 186/1644 objects degraded (11.314%), 23 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:10.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:10 smithi150 ceph-mon[52803]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-27T06:32:10.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:10 smithi150 ceph-mon[52803]: osdmap e128: 8 total, 7 up, 8 in 2024-02-27T06:32:10.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:10 smithi150 ceph-mon[52803]: from='osd.5 [v2:172.21.15.150:6808/2623515039,v1:172.21.15.150:6809/2623515039]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:32:10.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:10 smithi150 ceph-mon[52803]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:32:10.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:10 smithi150 ceph-mon[52803]: from='osd.5 ' entity='osd.5' 2024-02-27T06:32:11.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 267741 9.82 MB/sec execute 529 sec latency 819.320 ms 2024-02-27T06:32:11.605 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:11 smithi131 ceph-mon[66259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:32:11.605 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:11 smithi131 ceph-mon[66259]: osd.5 [v2:172.21.15.150:6808/2623515039,v1:172.21.15.150:6809/2623515039] boot 2024-02-27T06:32:11.605 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:11 smithi131 ceph-mon[66259]: osdmap e129: 8 total, 8 up, 8 in 2024-02-27T06:32:11.605 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:32:11.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:11 smithi150 ceph-mon[52803]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:32:11.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:11 smithi150 ceph-mon[52803]: osd.5 [v2:172.21.15.150:6808/2623515039,v1:172.21.15.150:6809/2623515039] boot 2024-02-27T06:32:11.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:11 smithi150 ceph-mon[52803]: osdmap e129: 8 total, 8 up, 8 in 2024-02-27T06:32:11.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-27T06:32:12.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:12 smithi131 ceph-mon[66259]: pgmap v230: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 347 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 477 op/s; 188/1647 objects degraded (11.415%) 2024-02-27T06:32:12.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:12 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:12.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:12 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:32:12.346 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:12 smithi131 ceph-mon[66259]: osdmap e130: 8 total, 8 up, 8 in 2024-02-27T06:32:12.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 267855 9.81 MB/sec execute 530 sec latency 1124.687 ms 2024-02-27T06:32:12.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:12 smithi150 ceph-mon[52803]: pgmap v230: 97 pgs: 7 active+undersized, 23 active+undersized+degraded, 67 active+clean; 347 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 477 op/s; 188/1647 objects degraded (11.415%) 2024-02-27T06:32:12.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:12 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:12.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:12 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:32:12.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:12 smithi150 ceph-mon[52803]: osdmap e130: 8 total, 8 up, 8 in 2024-02-27T06:32:13.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 268118 9.80 MB/sec execute 531 sec latency 981.114 ms 2024-02-27T06:32:13.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:13 smithi131 ceph-mon[66259]: pgmap v233: 97 pgs: 5 peering, 6 active+undersized, 19 active+undersized+degraded, 67 active+clean; 347 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.8 MiB/s wr, 413 op/s; 145/1626 objects degraded (8.918%) 2024-02-27T06:32:13.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:13 smithi131 ceph-mon[66259]: osdmap e131: 8 total, 8 up, 8 in 2024-02-27T06:32:14.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:13 smithi150 ceph-mon[52803]: pgmap v233: 97 pgs: 5 peering, 6 active+undersized, 19 active+undersized+degraded, 67 active+clean; 347 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.8 MiB/s wr, 413 op/s; 145/1626 objects degraded (8.918%) 2024-02-27T06:32:14.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:13 smithi150 ceph-mon[52803]: osdmap e131: 8 total, 8 up, 8 in 2024-02-27T06:32:14.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 268352 9.80 MB/sec execute 532 sec latency 647.767 ms 2024-02-27T06:32:14.957 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:14 smithi131 ceph-mon[66259]: osdmap e132: 8 total, 8 up, 8 in 2024-02-27T06:32:15.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:14 smithi150 ceph-mon[52803]: osdmap e132: 8 total, 8 up, 8 in 2024-02-27T06:32:15.559 INFO:teuthology.orchestra.run.smithi131.stdout: 5 268621 9.79 MB/sec execute 533 sec latency 885.990 ms 2024-02-27T06:32:15.991 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:15 smithi131 ceph-mon[66259]: pgmap v236: 97 pgs: 1 remapped+peering, 6 peering, 3 active+undersized, 3 active+undersized+degraded, 84 active+clean; 341 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 155 op/s; 18/1626 objects degraded (1.107%); 384 KiB/s, 24 objects/s recovering 2024-02-27T06:32:15.991 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:15 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 18/1626 objects degraded (1.107%), 3 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:15.991 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:15 smithi131 ceph-mon[66259]: osdmap e133: 8 total, 8 up, 8 in 2024-02-27T06:32:16.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:15 smithi150 ceph-mon[52803]: pgmap v236: 97 pgs: 1 remapped+peering, 6 peering, 3 active+undersized, 3 active+undersized+degraded, 84 active+clean; 341 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 155 op/s; 18/1626 objects degraded (1.107%); 384 KiB/s, 24 objects/s recovering 2024-02-27T06:32:16.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:15 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 18/1626 objects degraded (1.107%), 3 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:16.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:15 smithi150 ceph-mon[52803]: osdmap e133: 8 total, 8 up, 8 in 2024-02-27T06:32:16.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 268914 9.78 MB/sec execute 534 sec latency 57.972 ms 2024-02-27T06:32:17.002 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:16 smithi131 ceph-mon[66259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/1626 objects degraded (1.107%), 3 pgs degraded) 2024-02-27T06:32:17.002 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:16 smithi131 ceph-mon[66259]: Cluster is now healthy 2024-02-27T06:32:17.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:16 smithi150 ceph-mon[52803]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/1626 objects degraded (1.107%), 3 pgs degraded) 2024-02-27T06:32:17.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:16 smithi150 ceph-mon[52803]: Cluster is now healthy 2024-02-27T06:32:17.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 269201 9.78 MB/sec execute 535 sec latency 52.597 ms 2024-02-27T06:32:18.019 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:17 smithi131 ceph-mon[66259]: pgmap v238: 97 pgs: 1 remapped+peering, 1 peering, 95 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.4 MiB/s wr, 191 op/s; 535 KiB/s, 35 objects/s recovering 2024-02-27T06:32:18.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:17 smithi150 ceph-mon[52803]: pgmap v238: 97 pgs: 1 remapped+peering, 1 peering, 95 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.4 MiB/s wr, 191 op/s; 535 KiB/s, 35 objects/s recovering 2024-02-27T06:32:18.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 269578 9.78 MB/sec execute 536 sec latency 54.249 ms 2024-02-27T06:32:19.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 269920 9.78 MB/sec execute 537 sec latency 42.356 ms 2024-02-27T06:32:19.973 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:19 smithi131 ceph-mon[66259]: pgmap v239: 97 pgs: 1 peering, 96 active+clean; 344 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 2.9 MiB/s wr, 329 op/s; 424 KiB/s, 33 objects/s recovering 2024-02-27T06:32:20.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:19 smithi150 ceph-mon[52803]: pgmap v239: 97 pgs: 1 peering, 96 active+clean; 344 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 2.9 MiB/s wr, 329 op/s; 424 KiB/s, 33 objects/s recovering 2024-02-27T06:32:20.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 270238 9.78 MB/sec execute 538 sec latency 60.089 ms 2024-02-27T06:32:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 270562 9.78 MB/sec execute 539 sec latency 38.217 ms 2024-02-27T06:32:22.019 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:21 smithi131 ceph-mon[66259]: pgmap v240: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 2.6 MiB/s wr, 248 op/s; 518 KiB/s, 23 objects/s recovering 2024-02-27T06:32:22.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:21 smithi150 ceph-mon[52803]: pgmap v240: 97 pgs: 97 active+clean; 340 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 2.6 MiB/s wr, 248 op/s; 518 KiB/s, 23 objects/s recovering 2024-02-27T06:32:22.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 270902 9.78 MB/sec execute 540 sec latency 48.920 ms 2024-02-27T06:32:23.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 271260 9.78 MB/sec execute 541 sec latency 35.625 ms 2024-02-27T06:32:24.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:23 smithi131 ceph-mon[66259]: pgmap v241: 97 pgs: 97 active+clean; 347 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.8 MiB/s wr, 419 op/s; 459 KiB/s, 21 objects/s recovering 2024-02-27T06:32:24.047 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:23 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T06:32:24.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:23 smithi150 ceph-mon[52803]: pgmap v241: 97 pgs: 97 active+clean; 347 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.8 MiB/s wr, 419 op/s; 459 KiB/s, 21 objects/s recovering 2024-02-27T06:32:24.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:23 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T06:32:24.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 271478 9.77 MB/sec execute 542 sec latency 47.895 ms 2024-02-27T06:32:25.102 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:24 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T06:32:25.102 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:24 smithi131 ceph-mon[66259]: Upgrade: osd.6 is safe to restart 2024-02-27T06:32:25.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:24 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-27T06:32:25.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:24 smithi150 ceph-mon[52803]: Upgrade: osd.6 is safe to restart 2024-02-27T06:32:25.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 271825 9.77 MB/sec execute 543 sec latency 49.040 ms 2024-02-27T06:32:26.376 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:26 smithi131 ceph-mon[66259]: pgmap v242: 97 pgs: 97 active+clean; 350 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 393 op/s; 286 KiB/s, 10 objects/s recovering 2024-02-27T06:32:26.376 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:26 smithi131 ceph-mon[66259]: Upgrade: Updating osd.6 2024-02-27T06:32:26.376 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:26.376 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T06:32:26.376 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:32:26.376 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:26 smithi131 ceph-mon[66259]: Deploying daemon osd.6 on smithi150 2024-02-27T06:32:26.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 272102 9.76 MB/sec execute 544 sec latency 62.128 ms 2024-02-27T06:32:26.610 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:26 smithi150 ceph-mon[52803]: pgmap v242: 97 pgs: 97 active+clean; 350 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.0 MiB/s wr, 393 op/s; 286 KiB/s, 10 objects/s recovering 2024-02-27T06:32:26.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:26 smithi150 ceph-mon[52803]: Upgrade: Updating osd.6 2024-02-27T06:32:26.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:26.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-27T06:32:26.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:32:26.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:26 smithi150 ceph-mon[52803]: Deploying daemon osd.6 on smithi150 2024-02-27T06:32:27.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 272551 9.77 MB/sec execute 545 sec latency 28.905 ms 2024-02-27T06:32:27.604 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:27 smithi150 ceph-mon[52803]: pgmap v243: 97 pgs: 97 active+clean; 346 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.7 MiB/s wr, 442 op/s; 243 KiB/s, 9 objects/s recovering 2024-02-27T06:32:27.604 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:27.604 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:32:27.604 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:27.641 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:27 smithi131 ceph-mon[66259]: pgmap v243: 97 pgs: 97 active+clean; 346 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.7 MiB/s wr, 442 op/s; 243 KiB/s, 9 objects/s recovering 2024-02-27T06:32:27.641 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:27.641 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:32:27.641 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:28.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 272927 9.77 MB/sec execute 546 sec latency 47.477 ms 2024-02-27T06:32:29.361 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:29 smithi150 systemd[1]: Stopping Ceph osd.6 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:32:29.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 273359 9.77 MB/sec execute 547 sec latency 44.240 ms 2024-02-27T06:32:29.695 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:29 smithi131 ceph-mon[66259]: pgmap v244: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.7 MiB/s wr, 500 op/s; 154 KiB/s, 2 objects/s recovering 2024-02-27T06:32:29.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:29 smithi150 ceph-mon[52803]: pgmap v244: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.7 MiB/s wr, 500 op/s; 154 KiB/s, 2 objects/s recovering 2024-02-27T06:32:29.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:29 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[36932]: 2024-02-27T06:32:29.556+0000 7f8444240700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:32:29.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:29 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[36932]: 2024-02-27T06:32:29.556+0000 7f8444240700 -1 osd.6 133 *** Got signal Terminated *** 2024-02-27T06:32:29.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:29 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[36932]: 2024-02-27T06:32:29.556+0000 7f8444240700 -1 osd.6 133 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:32:30.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 273647 9.77 MB/sec execute 548 sec latency 44.663 ms 2024-02-27T06:32:30.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:30 smithi131 ceph-mon[66259]: osd.6 marked itself down and dead 2024-02-27T06:32:30.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:30 smithi150 ceph-mon[52803]: osd.6 marked itself down and dead 2024-02-27T06:32:31.492 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:31 smithi150 podman[61802]: 2024-02-27 06:32:31.190582399 +0000 UTC m=+1.764167165 container died a2b742f0d0bfafa4f5d5c69da92e1e58bb9dad2f5ba5e52d5c05e586ab18b011 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.buildah.version=1.19.8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , version=8, io.k8s.display-name=CentOS Stream 8, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.expose-services=) 2024-02-27T06:32:31.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 273648 9.75 MB/sec execute 549 sec latency 1033.211 ms 2024-02-27T06:32:31.784 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:31 smithi131 ceph-mon[66259]: pgmap v245: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 5.1 MiB/s wr, 538 op/s; 154 KiB/s, 0 objects/s recovering 2024-02-27T06:32:31.784 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:31 smithi131 ceph-mon[66259]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:32:31.784 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:31 smithi131 ceph-mon[66259]: osdmap e134: 8 total, 7 up, 8 in 2024-02-27T06:32:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:31 smithi150 ceph-mon[52803]: pgmap v245: 97 pgs: 97 active+clean; 351 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 5.1 MiB/s wr, 538 op/s; 154 KiB/s, 0 objects/s recovering 2024-02-27T06:32:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:31 smithi150 ceph-mon[52803]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:32:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:31 smithi150 ceph-mon[52803]: osdmap e134: 8 total, 7 up, 8 in 2024-02-27T06:32:31.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:31 smithi150 podman[61802]: 2024-02-27 06:32:31.67386525 +0000 UTC m=+2.247450024 container cleanup a2b742f0d0bfafa4f5d5c69da92e1e58bb9dad2f5ba5e52d5c05e586ab18b011 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, distribution-scope=public, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_CLEAN=True, architecture=x86_64, vcs-type=git, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True) 2024-02-27T06:32:31.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:31 smithi150 bash[61802]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6 2024-02-27T06:32:32.348 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:31 smithi150 podman[61820]: 2024-02-27 06:32:31.959040563 +0000 UTC m=+0.768816382 container remove a2b742f0d0bfafa4f5d5c69da92e1e58bb9dad2f5ba5e52d5c05e586ab18b011 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, ceph=True, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, name=centos-stream, com.redhat.component=centos-stream-container, release=754, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8) 2024-02-27T06:32:32.555 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:32 smithi131 ceph-mon[66259]: osdmap e135: 8 total, 7 up, 8 in 2024-02-27T06:32:32.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 273961 9.75 MB/sec execute 550 sec latency 1046.535 ms 2024-02-27T06:32:32.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:32 smithi150 ceph-mon[52803]: osdmap e135: 8 total, 7 up, 8 in 2024-02-27T06:32:32.611 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:32 smithi150 podman[61907]: 2024-02-27 06:32:32.242551537 +0000 UTC m=+0.019509019 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:32:32.611 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:32 smithi150 podman[61907]: 2024-02-27 06:32:32.501797689 +0000 UTC m=+0.278755151 container create 9e5475f82c6d086b936f9a405b1bec33193c3228bdea86f48620b9245e5e2fdb (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:32:33.332 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:32:33.361 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:33 smithi150 podman[61907]: 2024-02-27 06:32:33.105311566 +0000 UTC m=+0.882269030 container init 9e5475f82c6d086b936f9a405b1bec33193c3228bdea86f48620b9245e5e2fdb (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-27T06:32:33.361 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:33 smithi150 podman[61907]: 2024-02-27 06:32:33.108692137 +0000 UTC m=+0.885649606 container start 9e5475f82c6d086b936f9a405b1bec33193c3228bdea86f48620b9245e5e2fdb (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:32:33.361 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:33 smithi150 podman[61907]: 2024-02-27 06:32:33.189027534 +0000 UTC m=+0.965985012 container attach 9e5475f82c6d086b936f9a405b1bec33193c3228bdea86f48620b9245e5e2fdb (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-27T06:32:33.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 274318 9.76 MB/sec execute 551 sec latency 40.339 ms 2024-02-27T06:32:33.582 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:33 smithi131 ceph-mon[66259]: pgmap v248: 97 pgs: 9 peering, 7 stale+active+clean, 81 active+clean; 352 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 5.4 MiB/s wr, 581 op/s 2024-02-27T06:32:33.582 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:33 smithi131 ceph-mon[66259]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-27T06:32:33.819 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:33 smithi150 ceph-mon[52803]: pgmap v248: 97 pgs: 9 peering, 7 stale+active+clean, 81 active+clean; 352 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 5.4 MiB/s wr, 581 op/s 2024-02-27T06:32:33.819 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:33 smithi150 ceph-mon[52803]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-27T06:32:33.819 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:33 smithi150 podman[61907]: 2024-02-27 06:32:33.438570168 +0000 UTC m=+1.215527631 container died 9e5475f82c6d086b936f9a405b1bec33193c3228bdea86f48620b9245e5e2fdb (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-27T06:32:34.310 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:32:34.310 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (8m) 107s ago 17m 22.0M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:32:34.310 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (4m) 107s ago 17m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:32:34.310 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (4m) 31s ago 17m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:32:34.310 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (8m) 107s ago 17m 51.9M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:32:34.310 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (11m) 107s ago 11m 39.1M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:32:34.310 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (11m) 31s ago 11m 15.2M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:32:34.310 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (7m) 107s ago 18m 508M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (6m) 31s ago 16m 451M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (6m) 107s ago 18m 49.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (5m) 31s ago 16m 42.0M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (11m) 107s ago 11m 277M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (11m) 31s ago 11m 83.9M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (8m) 107s ago 17m 17.9M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (7m) 31s ago 16m 17.2M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (4m) 107s ago 16m 334M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (3m) 107s ago 15m 285M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (2m) 107s ago 15m 199M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (119s) 107s ago 14m 14.9M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 3d240baa2aed 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (74s) 31s ago 14m 267M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 7dc53922c309 2024-02-27T06:32:34.311 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (36s) 31s ago 14m 11.6M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 6865822553f8 2024-02-27T06:32:34.312 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (13m) 31s ago 13m 565M 2704M 17.2.0 e1d6a67b021e a2b742f0d0bf 2024-02-27T06:32:34.312 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (13m) 31s ago 13m 603M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:32:34.312 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (6m) 107s ago 16m 68.0M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:32:34.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 274672 9.76 MB/sec execute 552 sec latency 50.394 ms 2024-02-27T06:32:34.607 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:34 smithi131 ceph-mon[66259]: from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:34.607 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:34 smithi131 ceph-mon[66259]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:34.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:34 smithi150 ceph-mon[52803]: from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:34.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:34 smithi150 ceph-mon[52803]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:34.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:34 smithi150 podman[61932]: 2024-02-27 06:32:34.411500153 +0000 UTC m=+0.967010738 container remove 9e5475f82c6d086b936f9a405b1bec33193c3228bdea86f48620b9245e5e2fdb (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-27T06:32:34.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:34 smithi150 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.6.service: Deactivated successfully. 2024-02-27T06:32:34.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:34 smithi150 systemd[1]: Stopped Ceph osd.6 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:32:34.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:34 smithi150 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.6.service: Consumed 5min 9.271s CPU time. 2024-02-27T06:32:34.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:34 smithi150 systemd[1]: Starting Ceph osd.6 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:32:34.951 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:32:34.951 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:32:34.951 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:32:34.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:32:34.951 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:32:34.951 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:32:34.951 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:32:34.951 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:32:34.951 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-27T06:32:34.951 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 6 2024-02-27T06:32:34.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:32:34.952 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:32:34.952 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:32:34.952 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:32:34.952 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:32:34.952 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-27T06:32:34.952 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 10 2024-02-27T06:32:34.952 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:32:34.952 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:32:35.191 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:34 smithi150 podman[62036]: 2024-02-27 06:32:34.780503071 +0000 UTC m=+0.017855047 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:32:35.191 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:34 smithi150 podman[62036]: 2024-02-27 06:32:34.912287746 +0000 UTC m=+0.149639731 container create a1b4f215bf7617120798cd457de74952e6c11e4b191417cdb68471c6b2d30da5 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222) 2024-02-27T06:32:35.191 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 podman[62036]: 2024-02-27 06:32:35.182320599 +0000 UTC m=+0.419672575 container init a1b4f215bf7617120798cd457de74952e6c11e4b191417cdb68471c6b2d30da5 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:32:35.456 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:32:35.456 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:32:35.456 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:32:35.456 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:32:35.456 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:32:35.456 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:32:35.456 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:32:35.456 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:32:35.456 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:32:35.456 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "12/23 daemons upgraded", 2024-02-27T06:32:35.456 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T06:32:35.457 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:32:35.457 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:32:35.550 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 podman[62036]: 2024-02-27 06:32:35.185773317 +0000 UTC m=+0.423125302 container start a1b4f215bf7617120798cd457de74952e6c11e4b191417cdb68471c6b2d30da5 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-27T06:32:35.550 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 podman[62036]: 2024-02-27 06:32:35.274129243 +0000 UTC m=+0.511481229 container attach a1b4f215bf7617120798cd457de74952e6c11e4b191417cdb68471c6b2d30da5 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:32:35.562 INFO:teuthology.orchestra.run.smithi131.stdout: 5 274947 9.75 MB/sec execute 553 sec latency 46.675 ms 2024-02-27T06:32:35.635 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:35 smithi131 ceph-mon[66259]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:35.635 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:35 smithi131 ceph-mon[66259]: pgmap v249: 97 pgs: 2 active+undersized, 10 active+undersized+degraded, 9 peering, 4 stale+active+clean, 72 active+clean; 349 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.1 MiB/s wr, 456 op/s; 85/1683 objects degraded (5.051%) 2024-02-27T06:32:35.635 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:35 smithi131 ceph-mon[66259]: Health check failed: Degraded data redundancy: 85/1683 objects degraded (5.051%), 10 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:35.635 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:35 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/2431654175' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:35.860 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:35 smithi150 ceph-mon[52803]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:35.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:35 smithi150 ceph-mon[52803]: pgmap v249: 97 pgs: 2 active+undersized, 10 active+undersized+degraded, 9 peering, 4 stale+active+clean, 72 active+clean; 349 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.1 MiB/s wr, 456 op/s; 85/1683 objects degraded (5.051%) 2024-02-27T06:32:35.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:35 smithi150 ceph-mon[52803]: Health check failed: Degraded data redundancy: 85/1683 objects degraded (5.051%), 10 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:35.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:35 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/2431654175' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:35.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate[62051]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-27T06:32:35.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate[62051]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-27T06:32:35.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 bash[62036]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-27T06:32:35.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 bash[62036]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-27T06:32:35.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 bash[62036]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-27T06:32:35.862 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 bash[62036]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-27T06:32:35.862 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 bash[62036]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-27T06:32:35.862 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 bash[62036]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-27T06:32:35.862 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 bash[62036]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-27T06:32:35.862 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate[62051]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-27T06:32:35.862 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate[62051]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-27T06:32:35.862 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate[62051]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-27T06:32:35.862 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate[62051]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-27T06:32:35.862 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate[62051]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-27T06:32:35.862 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:35 smithi150 podman[62036]: 2024-02-27 06:32:35.691111495 +0000 UTC m=+0.928463479 container died a1b4f215bf7617120798cd457de74952e6c11e4b191417cdb68471c6b2d30da5 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-27T06:32:36.107 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 1 pg inactive, 1 pg peering; Degraded data redundancy: 85/1683 objects degraded (5.051%), 10 pgs degraded 2024-02-27T06:32:36.107 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-27T06:32:36.107 INFO:teuthology.orchestra.run.smithi131.stdout: osd.6 (root=default,host=smithi150) is down 2024-02-27T06:32:36.107 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering 2024-02-27T06:32:36.107 INFO:teuthology.orchestra.run.smithi131.stdout: pg 4.a is stuck peering for 11m, current state peering, last acting [2,7] 2024-02-27T06:32:36.107 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 85/1683 objects degraded (5.051%), 10 pgs degraded 2024-02-27T06:32:36.107 INFO:teuthology.orchestra.run.smithi131.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-02-27T06:32:36.107 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-02-27T06:32:36.107 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.7 is active+undersized+degraded, acting [7,2] 2024-02-27T06:32:36.107 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-02-27T06:32:36.108 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-02-27T06:32:36.108 INFO:teuthology.orchestra.run.smithi131.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2024-02-27T06:32:36.108 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-02-27T06:32:36.108 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-02-27T06:32:36.108 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-02-27T06:32:36.108 INFO:teuthology.orchestra.run.smithi131.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-02-27T06:32:36.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 275254 9.74 MB/sec execute 554 sec latency 54.374 ms 2024-02-27T06:32:36.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:36 smithi150 ceph-mon[52803]: from='client.44315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:36.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:36 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/1358063605' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:32:36.611 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:36 smithi150 podman[62183]: 2024-02-27 06:32:36.331909764 +0000 UTC m=+0.634973070 container remove a1b4f215bf7617120798cd457de74952e6c11e4b191417cdb68471c6b2d30da5 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:32:36.611 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:36 smithi150 podman[62245]: 2024-02-27 06:32:36.44833417 +0000 UTC m=+0.018698528 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:32:36.611 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:36 smithi150 podman[62245]: 2024-02-27 06:32:36.554761064 +0000 UTC m=+0.125125400 container create 1dedb9e4ac7963fe7d1ed84c9f62077919df509000b9232966ebb19ea3aa2744 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-27T06:32:36.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:36 smithi131 ceph-mon[66259]: from='client.44315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:32:36.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:36 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/1358063605' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:32:37.111 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:36 smithi150 podman[62245]: 2024-02-27 06:32:36.807915726 +0000 UTC m=+0.378280072 container init 1dedb9e4ac7963fe7d1ed84c9f62077919df509000b9232966ebb19ea3aa2744 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-27T06:32:37.111 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:36 smithi150 podman[62245]: 2024-02-27 06:32:36.81105888 +0000 UTC m=+0.381423216 container start 1dedb9e4ac7963fe7d1ed84c9f62077919df509000b9232966ebb19ea3aa2744 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0) 2024-02-27T06:32:37.111 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:36 smithi150 ceph-osd[62264]: -- 172.21.15.150:0/3013081607 <== mon.1 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564ec681f680 con 0x564ec67e8c00 2024-02-27T06:32:37.111 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:36 smithi150 bash[62245]: 1dedb9e4ac7963fe7d1ed84c9f62077919df509000b9232966ebb19ea3aa2744 2024-02-27T06:32:37.111 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:36 smithi150 systemd[1]: Started Ceph osd.6 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:32:37.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 275625 9.74 MB/sec execute 555 sec latency 50.083 ms 2024-02-27T06:32:37.692 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:37 smithi131 ceph-mon[66259]: pgmap v250: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 355 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 5.2 MiB/s wr, 532 op/s; 205/1686 objects degraded (12.159%) 2024-02-27T06:32:37.692 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:37 smithi131 ceph-mon[66259]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-27T06:32:37.692 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:37.692 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:37.692 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:32:37.861 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:37 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:32:37.356+0000 7efd39d667c0 -1 Falling back to public interface 2024-02-27T06:32:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:37 smithi150 ceph-mon[52803]: pgmap v250: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 355 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 5.2 MiB/s wr, 532 op/s; 205/1686 objects degraded (12.159%) 2024-02-27T06:32:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:37 smithi150 ceph-mon[52803]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-27T06:32:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:32:38.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 275952 9.74 MB/sec execute 556 sec latency 48.645 ms 2024-02-27T06:32:39.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 276252 9.74 MB/sec execute 557 sec latency 83.057 ms 2024-02-27T06:32:39.741 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:39 smithi131 ceph-mon[66259]: pgmap v251: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.0 MiB/s wr, 359 op/s; 208/1701 objects degraded (12.228%) 2024-02-27T06:32:39.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:39 smithi150 ceph-mon[52803]: pgmap v251: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.0 MiB/s wr, 359 op/s; 208/1701 objects degraded (12.228%) 2024-02-27T06:32:40.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 276619 9.74 MB/sec execute 558 sec latency 47.863 ms 2024-02-27T06:32:40.758 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:40 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 208/1701 objects degraded (12.228%), 25 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:40.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:40 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 208/1701 objects degraded (12.228%), 25 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:41.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 277030 9.74 MB/sec execute 559 sec latency 33.921 ms 2024-02-27T06:32:42.029 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:41 smithi131 ceph-mon[66259]: pgmap v252: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.1 MiB/s wr, 330 op/s; 199/1656 objects degraded (12.017%) 2024-02-27T06:32:42.029 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:42.029 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:42.029 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:42.029 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:32:42.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:41 smithi150 ceph-mon[52803]: pgmap v252: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 348 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.1 MiB/s wr, 330 op/s; 199/1656 objects degraded (12.017%) 2024-02-27T06:32:42.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:42.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:42.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:42.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:32:42.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 277348 9.74 MB/sec execute 560 sec latency 44.655 ms 2024-02-27T06:32:43.340 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:43 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:43.340 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:43 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:43.340 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:43 smithi150 ceph-mon[52803]: pgmap v253: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.2 MiB/s wr, 427 op/s; 194/1644 objects degraded (11.800%); 0 B/s, 0 objects/s recovering 2024-02-27T06:32:43.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:43 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:43.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:43 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:43.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:43 smithi131 ceph-mon[66259]: pgmap v253: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 354 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.2 MiB/s wr, 427 op/s; 194/1644 objects degraded (11.800%); 0 B/s, 0 objects/s recovering 2024-02-27T06:32:43.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 277708 9.75 MB/sec execute 561 sec latency 38.711 ms 2024-02-27T06:32:44.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 278055 9.74 MB/sec execute 562 sec latency 63.448 ms 2024-02-27T06:32:45.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 278343 9.74 MB/sec execute 563 sec latency 57.527 ms 2024-02-27T06:32:45.576 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:45 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:32:45.324+0000 7efd39d667c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-02-27T06:32:45.642 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:45 smithi131 ceph-mon[66259]: pgmap v254: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.2 MiB/s wr, 411 op/s; 195/1641 objects degraded (11.883%); 0 B/s, 0 objects/s recovering 2024-02-27T06:32:45.642 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:45 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 195/1641 objects degraded (11.883%), 25 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:45.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:45 smithi150 ceph-mon[52803]: pgmap v254: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 350 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.2 MiB/s wr, 411 op/s; 195/1641 objects degraded (11.883%); 0 B/s, 0 objects/s recovering 2024-02-27T06:32:45.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:45 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 195/1641 objects degraded (11.883%), 25 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:46.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 278570 9.73 MB/sec execute 564 sec latency 44.900 ms 2024-02-27T06:32:47.111 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:46 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:32:46.738+0000 7efd39d667c0 -1 osd.6 133 log_to_monitors true 2024-02-27T06:32:47.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 278961 9.73 MB/sec execute 565 sec latency 56.885 ms 2024-02-27T06:32:47.959 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: pgmap v255: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 346 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.3 MiB/s wr, 524 op/s; 203/1635 objects degraded (12.416%) 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='osd.6 [v2:172.21.15.150:6816/408768190,v1:172.21.15.150:6817/408768190]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:47.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T06:32:47.961 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T06:32:47.961 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:47 smithi131 ceph-mon[66259]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-27T06:32:48.111 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:32:47.764+0000 7efd2b1be700 -1 osd.6 133 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:32:48.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: pgmap v255: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 346 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.3 MiB/s wr, 524 op/s; 203/1635 objects degraded (12.416%) 2024-02-27T06:32:48.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:48.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:48.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:32:48.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:32:48.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:48.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='osd.6 [v2:172.21.15.150:6816/408768190,v1:172.21.15.150:6817/408768190]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T06:32:48.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-27T06:32:48.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:48.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:32:48.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:48.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:48.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:32:48.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T06:32:48.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T06:32:48.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:47 smithi150 ceph-mon[52803]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-27T06:32:48.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 279323 9.72 MB/sec execute 566 sec latency 53.003 ms 2024-02-27T06:32:48.970 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:48 smithi131 ceph-mon[66259]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-27T06:32:48.970 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:48 smithi131 ceph-mon[66259]: osdmap e136: 8 total, 7 up, 8 in 2024-02-27T06:32:48.970 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:48 smithi131 ceph-mon[66259]: from='osd.6 [v2:172.21.15.150:6816/408768190,v1:172.21.15.150:6817/408768190]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:32:48.970 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:48 smithi131 ceph-mon[66259]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:32:49.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:48 smithi150 ceph-mon[52803]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-27T06:32:49.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:48 smithi150 ceph-mon[52803]: osdmap e136: 8 total, 7 up, 8 in 2024-02-27T06:32:49.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:48 smithi150 ceph-mon[52803]: from='osd.6 [v2:172.21.15.150:6816/408768190,v1:172.21.15.150:6817/408768190]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:32:49.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:48 smithi150 ceph-mon[52803]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:32:49.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 279448 9.71 MB/sec execute 567 sec latency 807.125 ms 2024-02-27T06:32:50.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:49 smithi131 ceph-mon[66259]: pgmap v257: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 351 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.3 MiB/s wr, 547 op/s; 210/1647 objects degraded (12.750%) 2024-02-27T06:32:50.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:49 smithi131 ceph-mon[66259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:32:50.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:49 smithi131 ceph-mon[66259]: osd.6 [v2:172.21.15.150:6816/408768190,v1:172.21.15.150:6817/408768190] boot 2024-02-27T06:32:50.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:49 smithi131 ceph-mon[66259]: osdmap e137: 8 total, 8 up, 8 in 2024-02-27T06:32:50.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:49 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:32:50.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:49 smithi150 ceph-mon[52803]: pgmap v257: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 351 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.3 MiB/s wr, 547 op/s; 210/1647 objects degraded (12.750%) 2024-02-27T06:32:50.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:49 smithi150 ceph-mon[52803]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:32:50.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:49 smithi150 ceph-mon[52803]: osd.6 [v2:172.21.15.150:6816/408768190,v1:172.21.15.150:6817/408768190] boot 2024-02-27T06:32:50.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:49 smithi150 ceph-mon[52803]: osdmap e137: 8 total, 8 up, 8 in 2024-02-27T06:32:50.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:49 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-27T06:32:50.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 279452 9.69 MB/sec execute 568 sec latency 1807.176 ms 2024-02-27T06:32:51.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:50 smithi131 ceph-mon[66259]: osdmap e138: 8 total, 8 up, 8 in 2024-02-27T06:32:51.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:50 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 210/1647 objects degraded (12.750%), 25 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:51.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:50 smithi131 ceph-mon[66259]: osdmap e139: 8 total, 8 up, 8 in 2024-02-27T06:32:51.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:50 smithi150 ceph-mon[52803]: osdmap e138: 8 total, 8 up, 8 in 2024-02-27T06:32:51.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:50 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 210/1647 objects degraded (12.750%), 25 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:51.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:50 smithi150 ceph-mon[52803]: osdmap e139: 8 total, 8 up, 8 in 2024-02-27T06:32:51.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 279706 9.70 MB/sec execute 569 sec latency 2110.083 ms 2024-02-27T06:32:51.789 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:51 smithi131 ceph-mon[66259]: pgmap v260: 97 pgs: 2 activating, 2 activating+degraded, 1 remapped+peering, 5 peering, 3 active+undersized, 17 active+undersized+degraded, 67 active+clean; 349 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 5.4 MiB/s wr, 522 op/s; 164/1635 objects degraded (10.031%); 0 B/s, 41 keys/s, 0 objects/s recovering 2024-02-27T06:32:52.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:51 smithi150 ceph-mon[52803]: pgmap v260: 97 pgs: 2 activating, 2 activating+degraded, 1 remapped+peering, 5 peering, 3 active+undersized, 17 active+undersized+degraded, 67 active+clean; 349 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 5.4 MiB/s wr, 522 op/s; 164/1635 objects degraded (10.031%); 0 B/s, 41 keys/s, 0 objects/s recovering 2024-02-27T06:32:52.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 279898 9.69 MB/sec execute 570 sec latency 650.912 ms 2024-02-27T06:32:52.846 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:52 smithi131 ceph-mon[66259]: osdmap e140: 8 total, 8 up, 8 in 2024-02-27T06:32:53.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:52 smithi150 ceph-mon[52803]: osdmap e140: 8 total, 8 up, 8 in 2024-02-27T06:32:53.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 280226 9.69 MB/sec execute 571 sec latency 886.580 ms 2024-02-27T06:32:53.891 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:53 smithi131 ceph-mon[66259]: pgmap v263: 97 pgs: 2 activating, 2 activating+degraded, 1 remapped+peering, 5 peering, 1 active+undersized, 4 active+recovery_wait+undersized+degraded+remapped, 13 active+undersized+degraded, 69 active+clean; 352 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s wr, 299 op/s; 179/1632 objects degraded (10.968%); 0 B/s, 62 keys/s, 0 objects/s recovering 2024-02-27T06:32:53.891 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:53 smithi131 ceph-mon[66259]: osdmap e141: 8 total, 8 up, 8 in 2024-02-27T06:32:54.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:53 smithi150 ceph-mon[52803]: pgmap v263: 97 pgs: 2 activating, 2 activating+degraded, 1 remapped+peering, 5 peering, 1 active+undersized, 4 active+recovery_wait+undersized+degraded+remapped, 13 active+undersized+degraded, 69 active+clean; 352 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.0 MiB/s wr, 299 op/s; 179/1632 objects degraded (10.968%); 0 B/s, 62 keys/s, 0 objects/s recovering 2024-02-27T06:32:54.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:53 smithi150 ceph-mon[52803]: osdmap e141: 8 total, 8 up, 8 in 2024-02-27T06:32:54.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 280562 9.69 MB/sec execute 572 sec latency 984.841 ms 2024-02-27T06:32:54.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:54 smithi131 ceph-mon[66259]: osdmap e142: 8 total, 8 up, 8 in 2024-02-27T06:32:55.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:54 smithi150 ceph-mon[52803]: osdmap e142: 8 total, 8 up, 8 in 2024-02-27T06:32:55.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 280962 9.70 MB/sec execute 573 sec latency 38.272 ms 2024-02-27T06:32:55.873 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:55 smithi131 ceph-mon[66259]: pgmap v266: 97 pgs: 2 activating, 2 activating+degraded, 1 remapped+peering, 2 peering, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 342 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 2.2 MiB/s wr, 282 op/s; 90/1575 objects degraded (5.714%); 1.3 MiB/s, 34 objects/s recovering 2024-02-27T06:32:55.873 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:55 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 90/1575 objects degraded (5.714%), 6 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:56.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:55 smithi150 ceph-mon[52803]: pgmap v266: 97 pgs: 2 activating, 2 activating+degraded, 1 remapped+peering, 2 peering, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 342 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 2.2 MiB/s wr, 282 op/s; 90/1575 objects degraded (5.714%); 1.3 MiB/s, 34 objects/s recovering 2024-02-27T06:32:56.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:55 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 90/1575 objects degraded (5.714%), 6 pgs degraded (PG_DEGRADED) 2024-02-27T06:32:56.563 INFO:teuthology.orchestra.run.smithi131.stdout: 5 281272 9.69 MB/sec execute 574 sec latency 42.784 ms 2024-02-27T06:32:57.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 281608 9.69 MB/sec execute 575 sec latency 43.417 ms 2024-02-27T06:32:57.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:57.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:32:57.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:57 smithi131 ceph-mon[66259]: pgmap v267: 97 pgs: 2 peering, 1 active+undersized+remapped, 94 active+clean; 347 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.3 MiB/s wr, 460 op/s; 3.9 MiB/s, 44 objects/s recovering 2024-02-27T06:32:57.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:32:57.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:32:57.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:57 smithi150 ceph-mon[52803]: pgmap v267: 97 pgs: 2 peering, 1 active+undersized+remapped, 94 active+clean; 347 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.3 MiB/s wr, 460 op/s; 3.9 MiB/s, 44 objects/s recovering 2024-02-27T06:32:58.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 281882 9.68 MB/sec execute 576 sec latency 53.830 ms 2024-02-27T06:32:58.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:58 smithi131 ceph-mon[66259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 90/1575 objects degraded (5.714%), 6 pgs degraded) 2024-02-27T06:32:58.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:58 smithi131 ceph-mon[66259]: Cluster is now healthy 2024-02-27T06:32:58.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:58 smithi150 ceph-mon[52803]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 90/1575 objects degraded (5.714%), 6 pgs degraded) 2024-02-27T06:32:58.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:58 smithi150 ceph-mon[52803]: Cluster is now healthy 2024-02-27T06:32:59.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 282193 9.68 MB/sec execute 577 sec latency 55.702 ms 2024-02-27T06:32:59.696 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:32:59 smithi131 ceph-mon[66259]: pgmap v268: 97 pgs: 97 active+clean; 346 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 3.6 MiB/s wr, 297 op/s; 3.3 MiB/s, 37 objects/s recovering 2024-02-27T06:32:59.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:32:59 smithi150 ceph-mon[52803]: pgmap v268: 97 pgs: 97 active+clean; 346 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 3.6 MiB/s wr, 297 op/s; 3.3 MiB/s, 37 objects/s recovering 2024-02-27T06:33:00.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 282473 9.68 MB/sec execute 578 sec latency 73.691 ms 2024-02-27T06:33:01.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 282768 9.67 MB/sec execute 579 sec latency 87.108 ms 2024-02-27T06:33:01.796 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:01 smithi131 ceph-mon[66259]: pgmap v269: 97 pgs: 97 active+clean; 348 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 4.8 MiB/s wr, 481 op/s; 2.8 MiB/s, 31 objects/s recovering 2024-02-27T06:33:01.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:01 smithi150 ceph-mon[52803]: pgmap v269: 97 pgs: 97 active+clean; 348 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 4.8 MiB/s wr, 481 op/s; 2.8 MiB/s, 31 objects/s recovering 2024-02-27T06:33:02.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 283050 9.66 MB/sec execute 580 sec latency 54.211 ms 2024-02-27T06:33:02.718 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:02 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T06:33:02.719 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:02 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T06:33:02.719 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:02 smithi131 ceph-mon[66259]: Upgrade: osd.7 is safe to restart 2024-02-27T06:33:02.812 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:02 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T06:33:02.812 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:02 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-27T06:33:02.812 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:02 smithi150 ceph-mon[52803]: Upgrade: osd.7 is safe to restart 2024-02-27T06:33:03.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 283261 9.66 MB/sec execute 581 sec latency 61.319 ms 2024-02-27T06:33:03.634 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:03 smithi131 ceph-mon[66259]: pgmap v270: 97 pgs: 97 active+clean; 345 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.4 MiB/s wr, 437 op/s; 1.8 MiB/s, 16 objects/s recovering 2024-02-27T06:33:03.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:03 smithi150 ceph-mon[52803]: pgmap v270: 97 pgs: 97 active+clean; 345 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.4 MiB/s wr, 437 op/s; 1.8 MiB/s, 16 objects/s recovering 2024-02-27T06:33:04.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 283598 9.66 MB/sec execute 582 sec latency 57.576 ms 2024-02-27T06:33:04.892 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:04 smithi131 ceph-mon[66259]: Upgrade: Updating osd.7 2024-02-27T06:33:04.892 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:04 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:04.892 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:04 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T06:33:04.892 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:04 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:33:04.893 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:04 smithi131 ceph-mon[66259]: Deploying daemon osd.7 on smithi150 2024-02-27T06:33:05.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:04 smithi150 ceph-mon[52803]: Upgrade: Updating osd.7 2024-02-27T06:33:05.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:04 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:05.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:04 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-27T06:33:05.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:04 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:33:05.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:04 smithi150 ceph-mon[52803]: Deploying daemon osd.7 on smithi150 2024-02-27T06:33:05.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 283932 9.66 MB/sec execute 583 sec latency 69.042 ms 2024-02-27T06:33:05.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:05 smithi131 ceph-mon[66259]: pgmap v271: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 425 op/s; 1.7 MiB/s, 15 objects/s recovering 2024-02-27T06:33:06.065 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:05 smithi150 ceph-mon[52803]: pgmap v271: 97 pgs: 97 active+clean; 343 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 425 op/s; 1.7 MiB/s, 15 objects/s recovering 2024-02-27T06:33:06.519 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:33:06.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 284244 9.66 MB/sec execute 584 sec latency 85.443 ms 2024-02-27T06:33:07.348 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:33:07.348 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (9m) 2m ago 18m 22.0M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:33:07.348 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (5m) 2m ago 18m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:33:07.348 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (5m) 26s ago 17m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:33:07.348 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (8m) 2m ago 17m 51.9M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:33:07.348 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (12m) 2m ago 12m 39.1M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (12m) 26s ago 12m 15.3M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (8m) 2m ago 19m 508M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (7m) 26s ago 17m 452M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (6m) 2m ago 19m 49.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (6m) 26s ago 17m 43.2M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (11m) 2m ago 11m 277M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (11m) 26s ago 11m 83.9M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (8m) 2m ago 17m 17.9M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (8m) 26s ago 17m 19.9M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (4m) 2m ago 16m 334M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (3m) 2m ago 16m 285M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (3m) 2m ago 15m 199M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (2m) 2m ago 15m 14.9M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 3d240baa2aed 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (107s) 26s ago 14m 289M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 7dc53922c309 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (69s) 26s ago 14m 271M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 6865822553f8 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (30s) 26s ago 14m 15.0M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 1dedb9e4ac79 2024-02-27T06:33:07.349 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (13m) 26s ago 13m 651M 2704M 17.2.0 e1d6a67b021e 0a13728d2a4d 2024-02-27T06:33:07.350 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (7m) 2m ago 17m 68.0M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:33:07.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 284507 9.66 MB/sec execute 585 sec latency 59.063 ms 2024-02-27T06:33:07.697 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:07 smithi131 ceph-mon[66259]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:07.697 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:07 smithi131 ceph-mon[66259]: pgmap v272: 97 pgs: 97 active+clean; 346 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.9 MiB/s wr, 476 op/s; 1.5 MiB/s, 9 objects/s recovering 2024-02-27T06:33:07.697 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:07 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:07.698 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:07 smithi131 ceph-mon[66259]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:07.821 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:33:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:33:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:33:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:33:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:33:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:33:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:33:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:33:07.821 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-27T06:33:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 7 2024-02-27T06:33:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:33:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:33:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:33:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:33:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:33:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-27T06:33:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 11 2024-02-27T06:33:07.822 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:33:07.822 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:33:07.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:07 smithi150 ceph-mon[52803]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:07.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:07 smithi150 ceph-mon[52803]: pgmap v272: 97 pgs: 97 active+clean; 346 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.9 MiB/s wr, 476 op/s; 1.5 MiB/s, 9 objects/s recovering 2024-02-27T06:33:07.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:07 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:07.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:07 smithi150 ceph-mon[52803]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:07.862 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:07 smithi150 systemd[1]: Stopping Ceph osd.7 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:33:08.208 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:33:08.208 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:33:08.208 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:33:08.209 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:33:08.209 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:33:08.209 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:33:08.209 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:33:08.209 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:33:08.209 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:33:08.209 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "13/23 daemons upgraded", 2024-02-27T06:33:08.209 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading osd daemons", 2024-02-27T06:33:08.209 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:33:08.209 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:33:08.361 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:07 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[40007]: 2024-02-27T06:33:07.926+0000 7f6e0e7f3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:33:08.361 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:07 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[40007]: 2024-02-27T06:33:07.926+0000 7f6e0e7f3700 -1 osd.7 142 *** Got signal Terminated *** 2024-02-27T06:33:08.362 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:07 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[40007]: 2024-02-27T06:33:07.926+0000 7f6e0e7f3700 -1 osd.7 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:33:08.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 284742 9.65 MB/sec execute 586 sec latency 61.883 ms 2024-02-27T06:33:08.709 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid 2024-02-27T06:33:08.709 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] OSD_DOWN: 1 osds down 2024-02-27T06:33:08.709 INFO:teuthology.orchestra.run.smithi131.stdout: osd.7 (root=default,host=smithi150) is down 2024-02-27T06:33:08.709 INFO:teuthology.orchestra.run.smithi131.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-02-27T06:33:08.709 INFO:teuthology.orchestra.run.smithi131.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-02-27T06:33:08.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:08 smithi131 ceph-mon[66259]: from='client.34438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:08.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:08 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/2262085467' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:08.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:08 smithi131 ceph-mon[66259]: osd.7 marked itself down and dead 2024-02-27T06:33:08.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:08 smithi131 ceph-mon[66259]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:09.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:08 smithi150 ceph-mon[52803]: from='client.34438 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:09.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:08 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/2262085467' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:09.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:08 smithi150 ceph-mon[52803]: osd.7 marked itself down and dead 2024-02-27T06:33:09.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:08 smithi150 ceph-mon[52803]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:09.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 284768 9.63 MB/sec execute 587 sec latency 891.402 ms 2024-02-27T06:33:09.686 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:09 smithi150 podman[64076]: 2024-02-27 06:33:09.382710847 +0000 UTC m=+1.594263722 container died 0a13728d2a4d6082d9cb46313a133abfd0baf7a3eb3e53cfac4bacd036ef6627 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., version=8, architecture=x86_64, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-27T06:33:09.941 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:09 smithi131 ceph-mon[66259]: pgmap v273: 97 pgs: 97 active+clean; 349 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.3 MiB/s wr, 459 op/s 2024-02-27T06:33:09.941 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:09 smithi131 ceph-mon[66259]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:33:09.941 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:09 smithi131 ceph-mon[66259]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-27T06:33:09.941 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:09 smithi131 ceph-mon[66259]: osdmap e143: 8 total, 7 up, 8 in 2024-02-27T06:33:09.941 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:09 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/665703158' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:33:10.047 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:09 smithi150 ceph-mon[52803]: pgmap v273: 97 pgs: 97 active+clean; 349 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.3 MiB/s wr, 459 op/s 2024-02-27T06:33:10.047 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:09 smithi150 ceph-mon[52803]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-27T06:33:10.047 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:09 smithi150 ceph-mon[52803]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-27T06:33:10.047 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:09 smithi150 ceph-mon[52803]: osdmap e143: 8 total, 7 up, 8 in 2024-02-27T06:33:10.047 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:09 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/665703158' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:33:10.047 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:09 smithi150 podman[64076]: 2024-02-27 06:33:09.767916643 +0000 UTC m=+1.979469523 container cleanup 0a13728d2a4d6082d9cb46313a133abfd0baf7a3eb3e53cfac4bacd036ef6627 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, architecture=x86_64, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, version=8) 2024-02-27T06:33:10.047 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:09 smithi150 bash[64076]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7 2024-02-27T06:33:10.361 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:10 smithi150 podman[64093]: 2024-02-27 06:33:10.036211034 +0000 UTC m=+0.653207438 container remove 0a13728d2a4d6082d9cb46313a133abfd0baf7a3eb3e53cfac4bacd036ef6627 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, distribution-scope=public, ceph=True, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, com.redhat.component=centos-stream-container, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-02-27T06:33:10.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 285080 9.63 MB/sec execute 588 sec latency 1049.023 ms 2024-02-27T06:33:10.695 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:10 smithi150 podman[64183]: 2024-02-27 06:33:10.310171831 +0000 UTC m=+0.023232749 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:33:10.695 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:10 smithi150 podman[64183]: 2024-02-27 06:33:10.58746421 +0000 UTC m=+0.300525120 container create a5c478933e1bef705354d1f25e4e9afe5ea67ed9b3653bdb07c93f55d231e8dc (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:33:10.697 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:10 smithi131 ceph-mon[66259]: osdmap e144: 8 total, 7 up, 8 in 2024-02-27T06:33:11.017 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:10 smithi150 ceph-mon[52803]: osdmap e144: 8 total, 7 up, 8 in 2024-02-27T06:33:11.017 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:10 smithi150 podman[64183]: 2024-02-27 06:33:10.907463661 +0000 UTC m=+0.620524573 container init a5c478933e1bef705354d1f25e4e9afe5ea67ed9b3653bdb07c93f55d231e8dc (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-02-27T06:33:11.017 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:10 smithi150 podman[64183]: 2024-02-27 06:33:10.911846479 +0000 UTC m=+0.624907395 container start a5c478933e1bef705354d1f25e4e9afe5ea67ed9b3653bdb07c93f55d231e8dc (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:33:11.017 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:11 smithi150 podman[64183]: 2024-02-27 06:33:11.007606532 +0000 UTC m=+0.720667446 container attach a5c478933e1bef705354d1f25e4e9afe5ea67ed9b3653bdb07c93f55d231e8dc (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True) 2024-02-27T06:33:11.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 285345 9.63 MB/sec execute 589 sec latency 46.230 ms 2024-02-27T06:33:11.605 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:11 smithi150 podman[64183]: 2024-02-27 06:33:11.272870531 +0000 UTC m=+0.985931444 container died a5c478933e1bef705354d1f25e4e9afe5ea67ed9b3653bdb07c93f55d231e8dc (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-02-27T06:33:11.727 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:11 smithi131 ceph-mon[66259]: pgmap v276: 97 pgs: 11 active+undersized+degraded, 8 peering, 5 stale+active+clean, 73 active+clean; 347 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 394 op/s; 111/1653 objects degraded (6.715%) 2024-02-27T06:33:11.727 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:11 smithi131 ceph-mon[66259]: Health check failed: Degraded data redundancy: 111/1653 objects degraded (6.715%), 11 pgs degraded (PG_DEGRADED) 2024-02-27T06:33:11.727 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:11.727 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:33:11.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:11 smithi150 ceph-mon[52803]: pgmap v276: 97 pgs: 11 active+undersized+degraded, 8 peering, 5 stale+active+clean, 73 active+clean; 347 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.0 MiB/s wr, 394 op/s; 111/1653 objects degraded (6.715%) 2024-02-27T06:33:11.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:11 smithi150 ceph-mon[52803]: Health check failed: Degraded data redundancy: 111/1653 objects degraded (6.715%), 11 pgs degraded (PG_DEGRADED) 2024-02-27T06:33:11.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:11.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:33:12.361 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:12 smithi150 podman[64209]: 2024-02-27 06:33:12.046030922 +0000 UTC m=+0.766213582 container remove a5c478933e1bef705354d1f25e4e9afe5ea67ed9b3653bdb07c93f55d231e8dc (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5) 2024-02-27T06:33:12.361 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:12 smithi150 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.7.service: Deactivated successfully. 2024-02-27T06:33:12.361 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:12 smithi150 systemd[1]: Stopped Ceph osd.7 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:33:12.362 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:12 smithi150 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.7.service: Consumed 7min 30.353s CPU time. 2024-02-27T06:33:12.362 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:12 smithi150 systemd[1]: Starting Ceph osd.7 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:33:12.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 285667 9.63 MB/sec execute 590 sec latency 54.337 ms 2024-02-27T06:33:12.827 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:12 smithi150 podman[64312]: 2024-02-27 06:33:12.429565719 +0000 UTC m=+0.023639428 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:33:12.827 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:12 smithi150 podman[64312]: 2024-02-27 06:33:12.564132608 +0000 UTC m=+0.158206301 container create f87be7a73798d3fe77e3fff266869d2d90d1bbf71fce03891c7b66985d8b9597 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:33:13.111 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:12 smithi150 podman[64312]: 2024-02-27 06:33:12.817497269 +0000 UTC m=+0.411570962 container init f87be7a73798d3fe77e3fff266869d2d90d1bbf71fce03891c7b66985d8b9597 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:33:13.111 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:12 smithi150 podman[64312]: 2024-02-27 06:33:12.821179385 +0000 UTC m=+0.415253089 container start f87be7a73798d3fe77e3fff266869d2d90d1bbf71fce03891c7b66985d8b9597 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-02-27T06:33:13.111 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:12 smithi150 podman[64312]: 2024-02-27 06:33:12.909157834 +0000 UTC m=+0.503231566 container attach f87be7a73798d3fe77e3fff266869d2d90d1bbf71fce03891c7b66985d8b9597 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:33:13.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 286034 9.63 MB/sec execute 591 sec latency 35.766 ms 2024-02-27T06:33:13.611 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate[64326]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate[64326]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 bash[64312]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 bash[64312]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 bash[64312]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 bash[64312]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 bash[64312]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 bash[64312]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 bash[64312]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate[64326]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate[64326]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate[64326]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate[64326]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate[64326]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-27T06:33:13.612 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 podman[64312]: 2024-02-27 06:33:13.357147744 +0000 UTC m=+0.951221448 container died f87be7a73798d3fe77e3fff266869d2d90d1bbf71fce03891c7b66985d8b9597 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:33:13.709 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:13 smithi131 ceph-mon[66259]: pgmap v277: 97 pgs: 5 active+undersized, 24 active+undersized+degraded, 8 peering, 1 stale+active+clean, 59 active+clean; 348 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.2 MiB/s wr, 381 op/s; 209/1644 objects degraded (12.713%) 2024-02-27T06:33:13.990 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:13 smithi150 ceph-mon[52803]: pgmap v277: 97 pgs: 5 active+undersized, 24 active+undersized+degraded, 8 peering, 1 stale+active+clean, 59 active+clean; 348 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.2 MiB/s wr, 381 op/s; 209/1644 objects degraded (12.713%) 2024-02-27T06:33:14.250 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:13 smithi150 podman[64442]: 2024-02-27 06:33:13.981120013 +0000 UTC m=+0.617513224 container remove f87be7a73798d3fe77e3fff266869d2d90d1bbf71fce03891c7b66985d8b9597 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-activate, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) 2024-02-27T06:33:14.250 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:14 smithi150 podman[64505]: 2024-02-27 06:33:14.111872724 +0000 UTC m=+0.020611711 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:33:14.250 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:14 smithi150 podman[64505]: 2024-02-27 06:33:14.240727389 +0000 UTC m=+0.149466360 container create 50595e9edf3211586d5b63ba8ce3df0782ff8fb39ccab6d4711cec1e16762fee (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:33:14.511 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:14 smithi150 podman[64505]: 2024-02-27 06:33:14.476850289 +0000 UTC m=+0.385589260 container init 50595e9edf3211586d5b63ba8ce3df0782ff8fb39ccab6d4711cec1e16762fee (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-02-27T06:33:14.511 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:14 smithi150 podman[64505]: 2024-02-27 06:33:14.480534143 +0000 UTC m=+0.389273184 container start 50595e9edf3211586d5b63ba8ce3df0782ff8fb39ccab6d4711cec1e16762fee (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:33:14.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 286410 9.63 MB/sec execute 592 sec latency 47.476 ms 2024-02-27T06:33:14.861 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:14 smithi150 ceph-osd[64524]: -- 172.21.15.150:0/2890089248 <== mon.1 v2:172.21.15.150:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eaf69db680 con 0x55eaf69a4c00 2024-02-27T06:33:14.861 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:14 smithi150 bash[64505]: 50595e9edf3211586d5b63ba8ce3df0782ff8fb39ccab6d4711cec1e16762fee 2024-02-27T06:33:14.861 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:14 smithi150 systemd[1]: Started Ceph osd.7 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:33:15.361 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:15 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:33:15.022+0000 7f1e798867c0 -1 Falling back to public interface 2024-02-27T06:33:15.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 286707 9.62 MB/sec execute 593 sec latency 42.311 ms 2024-02-27T06:33:15.823 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:15 smithi131 ceph-mon[66259]: pgmap v278: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 349 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.4 MiB/s wr, 383 op/s; 267/1662 objects degraded (16.065%) 2024-02-27T06:33:15.823 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:15 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:15.824 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:15 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:15.824 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:15 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:33:15.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:15 smithi150 ceph-mon[52803]: pgmap v278: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 349 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.4 MiB/s wr, 383 op/s; 267/1662 objects degraded (16.065%) 2024-02-27T06:33:15.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:15 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:15.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:15 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:15.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:15 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:33:16.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 287068 9.62 MB/sec execute 594 sec latency 64.378 ms 2024-02-27T06:33:16.842 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:16 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 273/1698 objects degraded (16.078%), 35 pgs degraded (PG_DEGRADED) 2024-02-27T06:33:17.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:16 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 273/1698 objects degraded (16.078%), 35 pgs degraded (PG_DEGRADED) 2024-02-27T06:33:17.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 287403 9.62 MB/sec execute 595 sec latency 46.704 ms 2024-02-27T06:33:17.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:17 smithi131 ceph-mon[66259]: pgmap v279: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 351 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.7 MiB/s wr, 434 op/s; 273/1698 objects degraded (16.078%) 2024-02-27T06:33:18.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:17 smithi150 ceph-mon[52803]: pgmap v279: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 351 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.7 MiB/s wr, 434 op/s; 273/1698 objects degraded (16.078%) 2024-02-27T06:33:18.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 287720 9.62 MB/sec execute 596 sec latency 67.774 ms 2024-02-27T06:33:19.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 288067 9.63 MB/sec execute 597 sec latency 60.150 ms 2024-02-27T06:33:19.931 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:19 smithi131 ceph-mon[66259]: pgmap v280: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 350 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 432 op/s; 272/1701 objects degraded (15.991%) 2024-02-27T06:33:19.931 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:19.931 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:19 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:20.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:19 smithi150 ceph-mon[52803]: pgmap v280: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 350 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 432 op/s; 272/1701 objects degraded (15.991%) 2024-02-27T06:33:20.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:20.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:19 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:20.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 288351 9.62 MB/sec execute 598 sec latency 58.362 ms 2024-02-27T06:33:21.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 288703 9.62 MB/sec execute 599 sec latency 41.141 ms 2024-02-27T06:33:21.959 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:21 smithi131 ceph-mon[66259]: pgmap v281: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.6 MiB/s wr, 427 op/s; 266/1668 objects degraded (15.947%) 2024-02-27T06:33:21.959 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:21 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:21.960 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:21 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:22.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:21 smithi150 ceph-mon[52803]: pgmap v281: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.6 MiB/s wr, 427 op/s; 266/1668 objects degraded (15.947%) 2024-02-27T06:33:22.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:21 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:22.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:21 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:22.564 INFO:teuthology.orchestra.run.smithi131.stdout: 5 cleanup 600 sec 2024-02-27T06:33:22.815 INFO:teuthology.orchestra.run.smithi131.stdout: 0 cleanup 600 sec 2024-02-27T06:33:22.815 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:33:22.815 INFO:teuthology.orchestra.run.smithi131.stdout: Operation Count AvgLat MaxLat 2024-02-27T06:33:22.815 INFO:teuthology.orchestra.run.smithi131.stdout: ---------------------------------------- 2024-02-27T06:33:22.815 INFO:teuthology.orchestra.run.smithi131.stdout: NTCreateX 185230 4.737 2570.752 2024-02-27T06:33:22.815 INFO:teuthology.orchestra.run.smithi131.stdout: Close 136025 0.134 7.972 2024-02-27T06:33:22.816 INFO:teuthology.orchestra.run.smithi131.stdout: Rename 7852 10.200 1107.135 2024-02-27T06:33:22.816 INFO:teuthology.orchestra.run.smithi131.stdout: Unlink 37459 7.415 2446.467 2024-02-27T06:33:22.816 INFO:teuthology.orchestra.run.smithi131.stdout: Qpathinfo 168030 1.164 2515.938 2024-02-27T06:33:22.816 INFO:teuthology.orchestra.run.smithi131.stdout: Qfileinfo 29289 0.008 1.404 2024-02-27T06:33:22.816 INFO:teuthology.orchestra.run.smithi131.stdout: Qfsinfo 30811 0.616 33.930 2024-02-27T06:33:22.816 INFO:teuthology.orchestra.run.smithi131.stdout: Sfileinfo 15127 13.099 2451.941 2024-02-27T06:33:22.816 INFO:teuthology.orchestra.run.smithi131.stdout: Find 64958 0.949 19.476 2024-02-27T06:33:22.816 INFO:teuthology.orchestra.run.smithi131.stdout: WriteX 91557 13.818 2532.527 2024-02-27T06:33:22.816 INFO:teuthology.orchestra.run.smithi131.stdout: ReadX 290527 0.010 2.937 2024-02-27T06:33:22.817 INFO:teuthology.orchestra.run.smithi131.stdout: LockX 602 0.404 7.850 2024-02-27T06:33:22.817 INFO:teuthology.orchestra.run.smithi131.stdout: UnlockX 602 0.336 4.571 2024-02-27T06:33:22.817 INFO:teuthology.orchestra.run.smithi131.stdout: Flush 13024 0.002 0.053 2024-02-27T06:33:22.817 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:33:22.817 INFO:teuthology.orchestra.run.smithi131.stdout:Throughput 9.61883 MB/sec 5 clients 5 procs max_latency=2570.758 ms 2024-02-27T06:33:22.819 DEBUG:teuthology.orchestra.run.smithi131:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-02-27T06:33:22.851 DEBUG:teuthology.orchestra.run.smithi131:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-02-27T06:33:22.962 DEBUG:teuthology.orchestra.run.smithi131:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-02-27T06:33:22.985 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:22 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 263/1662 objects degraded (15.824%), 35 pgs degraded (PG_DEGRADED) 2024-02-27T06:33:22.989 INFO:teuthology.orchestra.run.smithi131.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-02-27T06:33:22.989 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:33:22.989 INFO:teuthology.orchestra.run.smithi131.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-02-27T06:33:22.990 INFO:teuthology.orchestra.run.smithi131.stdout:3 of 5 processes prepared for launch 0 sec 2024-02-27T06:33:23.090 INFO:teuthology.orchestra.run.smithi131.stdout:5 of 5 processes prepared for launch 0 sec 2024-02-27T06:33:23.090 INFO:teuthology.orchestra.run.smithi131.stdout:releasing clients 2024-02-27T06:33:23.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:22 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 263/1662 objects degraded (15.824%), 35 pgs degraded (PG_DEGRADED) 2024-02-27T06:33:24.008 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:23 smithi131 ceph-mon[66259]: pgmap v282: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 344 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.6 MiB/s wr, 449 op/s; 263/1662 objects degraded (15.824%) 2024-02-27T06:33:24.092 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:23 smithi150 ceph-mon[52803]: pgmap v282: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 344 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.6 MiB/s wr, 449 op/s; 263/1662 objects degraded (15.824%) 2024-02-27T06:33:25.090 INFO:teuthology.orchestra.run.smithi131.stdout: 5 478 33.81 MB/sec execute 1 sec latency 132.922 ms 2024-02-27T06:33:25.561 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:25.561 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:25.561 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:33:25.561 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:33:25.561 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:25.561 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:25.561 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: pgmap v283: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 358 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.6 MiB/s wr, 543 op/s; 270/1740 objects degraded (15.517%) 2024-02-27T06:33:25.561 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:33:25.561 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: Upgrade: Setting container_image for all osd 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-27T06:33:25.562 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-27T06:33:25.563 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-27T06:33:25.563 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-27T06:33:25.563 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-27T06:33:25.563 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-27T06:33:25.563 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-27T06:33:25.563 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-27T06:33:25.563 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: Upgrade: Setting require_osd_release to 19 squid 2024-02-27T06:33:25.563 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:25 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-27T06:33:25.861 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:33:25.572+0000 7f1e798867c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-02-27T06:33:25.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: pgmap v283: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 358 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.6 MiB/s wr, 543 op/s; 270/1740 objects degraded (15.517%) 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: Upgrade: Setting container_image for all osd 2024-02-27T06:33:25.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-27T06:33:25.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-27T06:33:25.864 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-27T06:33:25.864 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-27T06:33:25.864 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: Upgrade: Setting require_osd_release to 19 squid 2024-02-27T06:33:25.864 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:25 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-27T06:33:26.090 INFO:teuthology.orchestra.run.smithi131.stdout: 5 696 28.12 MB/sec execute 2 sec latency 51.058 ms 2024-02-27T06:33:26.641 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:26 smithi131 ceph-mon[66259]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-27T06:33:26.641 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-27T06:33:26.642 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:26 smithi131 ceph-mon[66259]: osdmap e145: 8 total, 7 up, 8 in 2024-02-27T06:33:26.642 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi131.mhuwwb"]}]: dispatch 2024-02-27T06:33:26.642 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:26 smithi131 ceph-mon[66259]: Upgrade: It appears safe to stop mds.foofs.smithi131.mhuwwb 2024-02-27T06:33:26.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:26 smithi150 ceph-mon[52803]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-27T06:33:26.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-27T06:33:26.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:26 smithi150 ceph-mon[52803]: osdmap e145: 8 total, 7 up, 8 in 2024-02-27T06:33:26.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi131.mhuwwb"]}]: dispatch 2024-02-27T06:33:26.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:26 smithi150 ceph-mon[52803]: Upgrade: It appears safe to stop mds.foofs.smithi131.mhuwwb 2024-02-27T06:33:27.090 INFO:teuthology.orchestra.run.smithi131.stdout: 5 944 19.70 MB/sec execute 3 sec latency 44.873 ms 2024-02-27T06:33:27.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:27.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:33:27.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:27 smithi131 ceph-mon[66259]: pgmap v285: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 365 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 12 MiB/s wr, 497 op/s; 254/1632 objects degraded (15.564%) 2024-02-27T06:33:27.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:27.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:33:27.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:27 smithi150 ceph-mon[52803]: pgmap v285: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 365 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 12 MiB/s wr, 497 op/s; 254/1632 objects degraded (15.564%) 2024-02-27T06:33:27.862 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:27 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:33:27.593+0000 7f1e798867c0 -1 osd.7 142 log_to_monitors true 2024-02-27T06:33:28.090 INFO:teuthology.orchestra.run.smithi131.stdout: 5 1310 17.86 MB/sec execute 4 sec latency 33.147 ms 2024-02-27T06:33:28.715 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:28 smithi131 ceph-mon[66259]: from='osd.7 [v2:172.21.15.150:6824/1876491304,v1:172.21.15.150:6825/1876491304]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T06:33:28.715 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:28 smithi131 ceph-mon[66259]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T06:33:28.715 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:28 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:28.715 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:28 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi131.mhuwwb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-27T06:33:28.715 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:28 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:33:28.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:28 smithi150 ceph-mon[52803]: from='osd.7 [v2:172.21.15.150:6824/1876491304,v1:172.21.15.150:6825/1876491304]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T06:33:28.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:28 smithi150 ceph-mon[52803]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-27T06:33:28.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:28 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:28.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:28 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi131.mhuwwb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-27T06:33:28.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:28 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:33:29.090 INFO:teuthology.orchestra.run.smithi131.stdout: 5 cleanup 5 sec 2024-02-27T06:33:29.295 INFO:teuthology.orchestra.run.smithi131.stdout:[1538] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000c99e0000001e' failed - Device or resource busy 2024-02-27T06:33:29.296 INFO:teuthology.orchestra.run.smithi131.stdout:[1538] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000c8b70000000c' failed - Device or resource busy 2024-02-27T06:33:29.296 INFO:teuthology.orchestra.run.smithi131.stdout:[1538] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000c8bc00000015' failed - Device or resource busy 2024-02-27T06:33:29.296 INFO:teuthology.orchestra.run.smithi131.stdout:[1538] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000c8b300000024' failed - Device or resource busy 2024-02-27T06:33:29.297 INFO:teuthology.orchestra.run.smithi131.stdout:[1538] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000c8c100000027' failed - Device or resource busy 2024-02-27T06:33:29.298 INFO:teuthology.orchestra.run.smithi131.stdout:[1538] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000c9a300000018' failed - Device or resource busy 2024-02-27T06:33:29.361 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:29 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:33:29.032+0000 7f1e66cd6700 -1 osd.7 142 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-27T06:33:29.367 INFO:teuthology.orchestra.run.smithi131.stdout: 0 cleanup 5 sec 2024-02-27T06:33:29.367 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:33:29.367 INFO:teuthology.orchestra.run.smithi131.stdout: Operation Count AvgLat MaxLat 2024-02-27T06:33:29.367 INFO:teuthology.orchestra.run.smithi131.stdout: ---------------------------------------- 2024-02-27T06:33:29.367 INFO:teuthology.orchestra.run.smithi131.stdout: NTCreateX 1066 5.895 113.220 2024-02-27T06:33:29.367 INFO:teuthology.orchestra.run.smithi131.stdout: Close 913 5.946 97.346 2024-02-27T06:33:29.367 INFO:teuthology.orchestra.run.smithi131.stdout: Rename 52 13.234 44.868 2024-02-27T06:33:29.368 INFO:teuthology.orchestra.run.smithi131.stdout: Unlink 106 5.726 89.319 2024-02-27T06:33:29.368 INFO:teuthology.orchestra.run.smithi131.stdout: Qpathinfo 711 1.742 89.724 2024-02-27T06:33:29.368 INFO:teuthology.orchestra.run.smithi131.stdout: Qfileinfo 372 15.125 109.209 2024-02-27T06:33:29.368 INFO:teuthology.orchestra.run.smithi131.stdout: Qfsinfo 165 0.411 4.375 2024-02-27T06:33:29.368 INFO:teuthology.orchestra.run.smithi131.stdout: Sfileinfo 170 14.393 31.259 2024-02-27T06:33:29.368 INFO:teuthology.orchestra.run.smithi131.stdout: Find 292 0.677 2.524 2024-02-27T06:33:29.368 INFO:teuthology.orchestra.run.smithi131.stdout: WriteX 1604 0.023 0.067 2024-02-27T06:33:29.368 INFO:teuthology.orchestra.run.smithi131.stdout: ReadX 1256 0.012 0.301 2024-02-27T06:33:29.368 INFO:teuthology.orchestra.run.smithi131.stdout: Flush 131 18.268 105.028 2024-02-27T06:33:29.368 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:33:29.368 INFO:teuthology.orchestra.run.smithi131.stdout:Throughput 17.8554 MB/sec 5 clients 5 procs max_latency=132.922 ms 2024-02-27T06:33:29.369 DEBUG:teuthology.parallel:result is None 2024-02-27T06:33:29.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:29 smithi131 ceph-mon[66259]: Upgrade: Updating mds.foofs.smithi131.mhuwwb 2024-02-27T06:33:29.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:29 smithi131 ceph-mon[66259]: Deploying daemon mds.foofs.smithi131.mhuwwb on smithi131 2024-02-27T06:33:29.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:29 smithi131 ceph-mon[66259]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-27T06:33:29.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:29 smithi131 ceph-mon[66259]: osdmap e146: 8 total, 7 up, 8 in 2024-02-27T06:33:29.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:29 smithi131 ceph-mon[66259]: pgmap v287: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 367 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 14 MiB/s wr, 585 op/s; 255/1626 objects degraded (15.683%) 2024-02-27T06:33:29.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:29 smithi131 ceph-mon[66259]: from='osd.7 [v2:172.21.15.150:6824/1876491304,v1:172.21.15.150:6825/1876491304]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:33:29.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:29 smithi131 ceph-mon[66259]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:33:29.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:29 smithi131 ceph-mon[66259]: from='osd.7 ' entity='osd.7' 2024-02-27T06:33:29.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:29 smithi150 ceph-mon[52803]: Upgrade: Updating mds.foofs.smithi131.mhuwwb 2024-02-27T06:33:29.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:29 smithi150 ceph-mon[52803]: Deploying daemon mds.foofs.smithi131.mhuwwb on smithi131 2024-02-27T06:33:29.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:29 smithi150 ceph-mon[52803]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-27T06:33:29.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:29 smithi150 ceph-mon[52803]: osdmap e146: 8 total, 7 up, 8 in 2024-02-27T06:33:29.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:29 smithi150 ceph-mon[52803]: pgmap v287: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 367 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 14 MiB/s wr, 585 op/s; 255/1626 objects degraded (15.683%) 2024-02-27T06:33:29.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:29 smithi150 ceph-mon[52803]: from='osd.7 [v2:172.21.15.150:6824/1876491304,v1:172.21.15.150:6825/1876491304]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:33:29.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:29 smithi150 ceph-mon[52803]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi150", "root=default"]}]: dispatch 2024-02-27T06:33:29.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:29 smithi150 ceph-mon[52803]: from='osd.7 ' entity='osd.7' 2024-02-27T06:33:30.628 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:30 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 255/1626 objects degraded (15.683%), 35 pgs degraded (PG_DEGRADED) 2024-02-27T06:33:30.628 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:30 smithi131 ceph-mon[66259]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:33:30.628 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:30 smithi131 ceph-mon[66259]: osd.7 [v2:172.21.15.150:6824/1876491304,v1:172.21.15.150:6825/1876491304] boot 2024-02-27T06:33:30.628 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:30 smithi131 ceph-mon[66259]: osdmap e147: 8 total, 8 up, 8 in 2024-02-27T06:33:30.628 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:33:30.628 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:30 smithi131 ceph-mon[66259]: osdmap e148: 8 total, 8 up, 8 in 2024-02-27T06:33:30.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:30 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 255/1626 objects degraded (15.683%), 35 pgs degraded (PG_DEGRADED) 2024-02-27T06:33:30.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:30 smithi150 ceph-mon[52803]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-27T06:33:30.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:30 smithi150 ceph-mon[52803]: osd.7 [v2:172.21.15.150:6824/1876491304,v1:172.21.15.150:6825/1876491304] boot 2024-02-27T06:33:30.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:30 smithi150 ceph-mon[52803]: osdmap e147: 8 total, 8 up, 8 in 2024-02-27T06:33:30.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-27T06:33:30.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:30 smithi150 ceph-mon[52803]: osdmap e148: 8 total, 8 up, 8 in 2024-02-27T06:33:31.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:30 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131[66255]: 2024-02-27T06:33:30.931+0000 7f1746787700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-27T06:33:31.361 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:33:31 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:33:31.022+0000 7f008dcf3700 -1 bdev(0x55955c8e2000 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 6 2024-02-27T06:33:31.361 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:33:31 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:33:31.018+0000 7f1e534af700 -1 bdev(0x55eaf77a0000 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-02-27T06:33:31.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:31 smithi131 ceph-mon[66259]: pgmap v290: 97 pgs: 8 active+undersized, 34 active+undersized+degraded, 55 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 16 MiB/s wr, 560 op/s; 212/1428 objects degraded (14.846%) 2024-02-27T06:33:31.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:31 smithi131 ceph-mon[66259]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-27T06:33:31.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:31 smithi131 ceph-mon[66259]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-27T06:33:31.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:31 smithi131 ceph-mon[66259]: osdmap e149: 8 total, 8 up, 8 in 2024-02-27T06:33:31.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:31 smithi131 ceph-mon[66259]: Standby daemon mds.foofs.smithi150.vsnjhu assigned to filesystem foofs as rank 0 2024-02-27T06:33:31.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:31 smithi131 ceph-mon[66259]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-27T06:33:31.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:31 smithi131 ceph-mon[66259]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-27T06:33:31.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:31 smithi131 ceph-mon[66259]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-02-27T06:33:31.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:31 smithi131 ceph-mon[66259]: fsmap foofs:1/1 {0=foofs.smithi150.vsnjhu=up:replay} 2024-02-27T06:33:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:31 smithi150 ceph-mon[52803]: pgmap v290: 97 pgs: 8 active+undersized, 34 active+undersized+degraded, 55 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 16 MiB/s wr, 560 op/s; 212/1428 objects degraded (14.846%) 2024-02-27T06:33:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:31 smithi150 ceph-mon[52803]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-27T06:33:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:31 smithi150 ceph-mon[52803]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-27T06:33:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:31 smithi150 ceph-mon[52803]: osdmap e149: 8 total, 8 up, 8 in 2024-02-27T06:33:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:31 smithi150 ceph-mon[52803]: Standby daemon mds.foofs.smithi150.vsnjhu assigned to filesystem foofs as rank 0 2024-02-27T06:33:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:31 smithi150 ceph-mon[52803]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-27T06:33:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:31 smithi150 ceph-mon[52803]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-27T06:33:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:31 smithi150 ceph-mon[52803]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-02-27T06:33:31.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:31 smithi150 ceph-mon[52803]: fsmap foofs:1/1 {0=foofs.smithi150.vsnjhu=up:replay} 2024-02-27T06:33:33.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:32 smithi131 ceph-mon[66259]: osdmap e150: 8 total, 8 up, 8 in 2024-02-27T06:33:33.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:32 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:33:33.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:32 smithi131 ceph-mon[66259]: mds.? [v2:172.21.15.150:6832/3371494173,v1:172.21.15.150:6833/3371494173] up:reconnect 2024-02-27T06:33:33.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:32 smithi131 ceph-mon[66259]: fsmap foofs:1/1 {0=foofs.smithi150.vsnjhu=up:reconnect} 2024-02-27T06:33:33.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:32 smithi150 ceph-mon[52803]: osdmap e150: 8 total, 8 up, 8 in 2024-02-27T06:33:33.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:32 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:33:33.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:32 smithi150 ceph-mon[52803]: mds.? [v2:172.21.15.150:6832/3371494173,v1:172.21.15.150:6833/3371494173] up:reconnect 2024-02-27T06:33:33.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:32 smithi150 ceph-mon[52803]: fsmap foofs:1/1 {0=foofs.smithi150.vsnjhu=up:reconnect} 2024-02-27T06:33:34.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:33 smithi131 ceph-mon[66259]: pgmap v293: 97 pgs: 4 remapped+peering, 3 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 22 active+undersized+degraded, 65 active+clean; 338 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 6.8 MiB/s wr, 213 op/s; 239/1482 objects degraded (16.127%); 1.0 MiB/s, 8 objects/s recovering 2024-02-27T06:33:34.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:33 smithi131 ceph-mon[66259]: reconnect by client.24335 172.21.15.150:0/1102554480 after 0.00099998 2024-02-27T06:33:34.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:33 smithi131 ceph-mon[66259]: reconnect by client.14538 172.21.15.131:0/2232015846 after 0.052999 2024-02-27T06:33:34.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:33 smithi131 ceph-mon[66259]: osdmap e151: 8 total, 8 up, 8 in 2024-02-27T06:33:34.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:33 smithi131 ceph-mon[66259]: mds.? [v2:172.21.15.150:6832/3371494173,v1:172.21.15.150:6833/3371494173] up:rejoin 2024-02-27T06:33:34.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:33 smithi131 ceph-mon[66259]: fsmap foofs:1/1 {0=foofs.smithi150.vsnjhu=up:rejoin} 2024-02-27T06:33:34.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:33 smithi131 ceph-mon[66259]: daemon mds.foofs.smithi150.vsnjhu is now active in filesystem foofs as rank 0 2024-02-27T06:33:34.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:33 smithi150 ceph-mon[52803]: pgmap v293: 97 pgs: 4 remapped+peering, 3 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 22 active+undersized+degraded, 65 active+clean; 338 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 6.8 MiB/s wr, 213 op/s; 239/1482 objects degraded (16.127%); 1.0 MiB/s, 8 objects/s recovering 2024-02-27T06:33:34.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:33 smithi150 ceph-mon[52803]: reconnect by client.24335 172.21.15.150:0/1102554480 after 0.00099998 2024-02-27T06:33:34.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:33 smithi150 ceph-mon[52803]: reconnect by client.14538 172.21.15.131:0/2232015846 after 0.052999 2024-02-27T06:33:34.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:33 smithi150 ceph-mon[52803]: osdmap e151: 8 total, 8 up, 8 in 2024-02-27T06:33:34.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:33 smithi150 ceph-mon[52803]: mds.? [v2:172.21.15.150:6832/3371494173,v1:172.21.15.150:6833/3371494173] up:rejoin 2024-02-27T06:33:34.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:33 smithi150 ceph-mon[52803]: fsmap foofs:1/1 {0=foofs.smithi150.vsnjhu=up:rejoin} 2024-02-27T06:33:34.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:33 smithi150 ceph-mon[52803]: daemon mds.foofs.smithi150.vsnjhu is now active in filesystem foofs as rank 0 2024-02-27T06:33:35.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:34 smithi131 ceph-mon[66259]: osdmap e152: 8 total, 8 up, 8 in 2024-02-27T06:33:35.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:34 smithi131 ceph-mon[66259]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-27T06:33:35.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:34 smithi131 ceph-mon[66259]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-27T06:33:35.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:34 smithi131 ceph-mon[66259]: mds.? [v2:172.21.15.150:6832/3371494173,v1:172.21.15.150:6833/3371494173] up:active 2024-02-27T06:33:35.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:34 smithi131 ceph-mon[66259]: mds.? [v2:172.21.15.131:6834/1153822240,v1:172.21.15.131:6835/1153822240] up:boot 2024-02-27T06:33:35.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:34 smithi131 ceph-mon[66259]: fsmap foofs:1 {0=foofs.smithi150.vsnjhu=up:active} 1 up:standby 2024-02-27T06:33:35.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:34 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:33:35.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:34 smithi150 ceph-mon[52803]: osdmap e152: 8 total, 8 up, 8 in 2024-02-27T06:33:35.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:34 smithi150 ceph-mon[52803]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-27T06:33:35.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:34 smithi150 ceph-mon[52803]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-27T06:33:35.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:34 smithi150 ceph-mon[52803]: mds.? [v2:172.21.15.150:6832/3371494173,v1:172.21.15.150:6833/3371494173] up:active 2024-02-27T06:33:35.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:34 smithi150 ceph-mon[52803]: mds.? [v2:172.21.15.131:6834/1153822240,v1:172.21.15.131:6835/1153822240] up:boot 2024-02-27T06:33:35.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:34 smithi150 ceph-mon[52803]: fsmap foofs:1 {0=foofs.smithi150.vsnjhu=up:active} 1 up:standby 2024-02-27T06:33:35.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:34 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:33:36.332 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:35 smithi131 ceph-mon[66259]: pgmap v296: 97 pgs: 1 peering, 2 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+undersized+degraded, 87 active+clean; 336 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 2.1 MiB/s wr, 166 op/s; 117/1473 objects degraded (7.943%); 8.0 MiB/s, 70 objects/s recovering 2024-02-27T06:33:36.332 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:35 smithi131 ceph-mon[66259]: Health check update: Degraded data redundancy: 117/1473 objects degraded (7.943%), 7 pgs degraded (PG_DEGRADED) 2024-02-27T06:33:36.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:35 smithi150 ceph-mon[52803]: pgmap v296: 97 pgs: 1 peering, 2 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+undersized+degraded, 87 active+clean; 336 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 2.1 MiB/s wr, 166 op/s; 117/1473 objects degraded (7.943%); 8.0 MiB/s, 70 objects/s recovering 2024-02-27T06:33:36.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:35 smithi150 ceph-mon[52803]: Health check update: Degraded data redundancy: 117/1473 objects degraded (7.943%), 7 pgs degraded (PG_DEGRADED) 2024-02-27T06:33:37.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:37 smithi131 ceph-mon[66259]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 117/1473 objects degraded (7.943%), 7 pgs degraded) 2024-02-27T06:33:37.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:37 smithi131 ceph-mon[66259]: Cluster is now healthy 2024-02-27T06:33:37.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:37 smithi150 ceph-mon[52803]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 117/1473 objects degraded (7.943%), 7 pgs degraded) 2024-02-27T06:33:37.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:37 smithi150 ceph-mon[52803]: Cluster is now healthy 2024-02-27T06:33:38.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:38 smithi131 ceph-mon[66259]: pgmap v297: 97 pgs: 1 peering, 96 active+clean; 272 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 41 MiB/s rd, 799 KiB/s wr, 145 op/s; 8.8 MiB/s, 79 objects/s recovering 2024-02-27T06:33:38.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:38 smithi150 ceph-mon[52803]: pgmap v297: 97 pgs: 1 peering, 96 active+clean; 272 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 41 MiB/s rd, 799 KiB/s wr, 145 op/s; 8.8 MiB/s, 79 objects/s recovering 2024-02-27T06:33:39.066 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:33:39.730 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:33:39.730 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (9m) 2m ago 18m 22.0M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:33:39.730 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (5m) 2m ago 18m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:33:39.730 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (5m) 20s ago 18m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:33:39.730 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (9m) 2m ago 18m 51.9M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:33:39.730 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (12m) 2m ago 12m 39.1M - 17.2.0 e1d6a67b021e 37a2aaaa8d26 2024-02-27T06:33:39.730 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (12m) 20s ago 12m 15.4M - 17.2.0 e1d6a67b021e 052b10582857 2024-02-27T06:33:39.730 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (8m) 2m ago 19m 508M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:33:39.730 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (7m) 20s ago 18m 452M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:33:39.730 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (7m) 2m ago 19m 49.1M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (6m) 20s ago 17m 44.7M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (12m) 2m ago 12m 277M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (12m) 20s ago 12m 83.9M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (9m) 2m ago 18m 17.9M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (8m) 20s ago 17m 20.0M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (5m) 2m ago 17m 334M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (4m) 2m ago 16m 285M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (3m) 2m ago 16m 199M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (3m) 2m ago 15m 14.9M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 3d240baa2aed 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (2m) 20s ago 15m 330M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 7dc53922c309 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (101s) 20s ago 15m 287M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 6865822553f8 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (63s) 20s ago 14m 277M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 1dedb9e4ac79 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (25s) 20s ago 14m 14.8M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 50595e9edf32 2024-02-27T06:33:39.731 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (7m) 2m ago 17m 68.0M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:33:40.138 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:33:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-27T06:33:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 12 2024-02-27T06:33:40.139 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:33:40.139 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:33:40.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:40 smithi131 ceph-mon[66259]: pgmap v298: 97 pgs: 97 active+clean; 266 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 560 KiB/s wr, 125 op/s; 7.5 MiB/s, 67 objects/s recovering 2024-02-27T06:33:40.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:40 smithi131 ceph-mon[66259]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:40.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:40 smithi150 ceph-mon[52803]: pgmap v298: 97 pgs: 97 active+clean; 266 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 35 MiB/s rd, 560 KiB/s wr, 125 op/s; 7.5 MiB/s, 67 objects/s recovering 2024-02-27T06:33:40.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:40 smithi150 ceph-mon[52803]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:40.498 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:33:40.498 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:33:40.498 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:33:40.499 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:33:40.499 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:33:40.499 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:33:40.499 INFO:teuthology.orchestra.run.smithi131.stdout: "osd", 2024-02-27T06:33:40.499 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:33:40.499 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:33:40.499 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:33:40.499 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "14/23 daemons upgraded", 2024-02-27T06:33:40.499 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading mds daemons", 2024-02-27T06:33:40.499 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:33:40.499 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:33:40.898 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:33:41.225 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:41 smithi131 ceph-mon[66259]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:41.225 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:41 smithi131 ceph-mon[66259]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:41.225 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:41 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/970541199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:41.225 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:41 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/335990750' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:33:41.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:41 smithi150 ceph-mon[52803]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:41.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:41 smithi150 ceph-mon[52803]: from='client.34470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:41.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:41 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/970541199' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:41.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:41 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/335990750' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:33:42.179 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:42 smithi131 ceph-mon[66259]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:42.180 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:42 smithi131 ceph-mon[66259]: pgmap v299: 97 pgs: 97 active+clean; 236 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 464 KiB/s wr, 121 op/s; 6.2 MiB/s, 55 objects/s recovering 2024-02-27T06:33:42.180 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:42.180 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:33:42.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:42 smithi150 ceph-mon[52803]: from='client.34478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:33:42.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:42 smithi150 ceph-mon[52803]: pgmap v299: 97 pgs: 97 active+clean; 236 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 464 KiB/s wr, 121 op/s; 6.2 MiB/s, 55 objects/s recovering 2024-02-27T06:33:42.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:42.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:33:43.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:43 smithi131 ceph-mon[66259]: daemon mds.foofs.smithi131.mhuwwb restarted 2024-02-27T06:33:43.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:43 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:43.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:43 smithi131 ceph-mon[66259]: osdmap e153: 8 total, 8 up, 8 in 2024-02-27T06:33:43.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:43 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:43.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:43 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:33:43.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:43 smithi150 ceph-mon[52803]: daemon mds.foofs.smithi131.mhuwwb restarted 2024-02-27T06:33:43.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:43 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:43.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:43 smithi150 ceph-mon[52803]: osdmap e153: 8 total, 8 up, 8 in 2024-02-27T06:33:43.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:43 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:43.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:43 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:33:44.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:44 smithi131 ceph-mon[66259]: pgmap v300: 97 pgs: 97 active+clean; 236 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 60 KiB/s wr, 53 op/s; 4.4 MiB/s, 46 objects/s recovering 2024-02-27T06:33:44.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:44 smithi131 ceph-mon[66259]: mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640] up:boot 2024-02-27T06:33:44.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:44 smithi131 ceph-mon[66259]: fsmap foofs:1 {0=foofs.smithi150.vsnjhu=up:active} 1 up:standby 2024-02-27T06:33:44.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:44 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:33:44.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:44 smithi150 ceph-mon[52803]: pgmap v300: 97 pgs: 97 active+clean; 236 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 60 KiB/s wr, 53 op/s; 4.4 MiB/s, 46 objects/s recovering 2024-02-27T06:33:44.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:44 smithi150 ceph-mon[52803]: mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640] up:boot 2024-02-27T06:33:44.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:44 smithi150 ceph-mon[52803]: fsmap foofs:1 {0=foofs.smithi150.vsnjhu=up:active} 1 up:standby 2024-02-27T06:33:44.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:44 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:33:46.331 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:46 smithi131 ceph-mon[66259]: pgmap v302: 97 pgs: 97 active+clean; 236 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 58 KiB/s wr, 49 op/s; 2.2 MiB/s, 19 objects/s recovering 2024-02-27T06:33:46.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:46 smithi150 ceph-mon[52803]: pgmap v302: 97 pgs: 97 active+clean; 236 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 58 KiB/s wr, 49 op/s; 2.2 MiB/s, 19 objects/s recovering 2024-02-27T06:33:47.947 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:47 smithi131 ceph-mon[66259]: pgmap v303: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 57 KiB/s wr, 22 op/s 2024-02-27T06:33:47.947 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:47 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:48.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:47 smithi150 ceph-mon[52803]: pgmap v303: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 57 KiB/s wr, 22 op/s 2024-02-27T06:33:48.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:47 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:49.519 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:49 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:49.519 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:49 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:49.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:49 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:49.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:49 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:50.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:50 smithi131 ceph-mon[66259]: pgmap v304: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 819 B/s rd, 23 KiB/s wr, 15 op/s 2024-02-27T06:33:50.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:50 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:50.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:50 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:50.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:50 smithi150 ceph-mon[52803]: pgmap v304: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 819 B/s rd, 23 KiB/s wr, 15 op/s 2024-02-27T06:33:50.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:50 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:50.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:50 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:51.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:51 smithi131 ceph-mon[66259]: pgmap v305: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 21 KiB/s wr, 1 op/s 2024-02-27T06:33:51.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:51 smithi150 ceph-mon[52803]: pgmap v305: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 21 KiB/s wr, 1 op/s 2024-02-27T06:33:54.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: pgmap v306: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 307 B/s wr, 0 op/s 2024-02-27T06:33:54.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: Detected new or changed devices on smithi131 2024-02-27T06:33:54.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:54.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:54.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:33:54.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:33:54.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:54.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:54.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:33:54.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:54.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:54.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:54.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:54.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi150.vsnjhu"]}]: dispatch 2024-02-27T06:33:54.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:54 smithi131 ceph-mon[66259]: Upgrade: It appears safe to stop mds.foofs.smithi150.vsnjhu 2024-02-27T06:33:54.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: pgmap v306: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 307 B/s wr, 0 op/s 2024-02-27T06:33:54.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: Detected new or changed devices on smithi131 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi150.vsnjhu"]}]: dispatch 2024-02-27T06:33:54.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:54 smithi150 ceph-mon[52803]: Upgrade: It appears safe to stop mds.foofs.smithi150.vsnjhu 2024-02-27T06:33:55.965 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:55 smithi150 ceph-mon[52803]: pgmap v307: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 174 B/s wr, 0 op/s 2024-02-27T06:33:55.965 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:55 smithi150 ceph-mon[52803]: Upgrade: Updating mds.foofs.smithi150.vsnjhu 2024-02-27T06:33:55.965 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:55 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:55.965 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:55 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi150.vsnjhu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-27T06:33:55.966 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:55 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:33:55.966 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:55 smithi150 ceph-mon[52803]: Deploying daemon mds.foofs.smithi150.vsnjhu on smithi150 2024-02-27T06:33:56.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:55 smithi131 ceph-mon[66259]: pgmap v307: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 174 B/s rd, 174 B/s wr, 0 op/s 2024-02-27T06:33:56.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:55 smithi131 ceph-mon[66259]: Upgrade: Updating mds.foofs.smithi150.vsnjhu 2024-02-27T06:33:56.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:55 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:33:56.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:55 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi150.vsnjhu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-27T06:33:56.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:55 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:33:56.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:55 smithi131 ceph-mon[66259]: Deploying daemon mds.foofs.smithi150.vsnjhu on smithi150 2024-02-27T06:33:56.790 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:56 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:33:57.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:56 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:33:57.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:56 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131[66255]: 2024-02-27T06:33:56.760+0000 7f1746787700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-27T06:33:57.106 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:33:56 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[75036]: 2024-02-27T06:33:56.778+0000 7f373237b700 -1 bdev(0x565261220000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 9 2024-02-27T06:33:57.106 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:33:56 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[75036]: 2024-02-27T06:33:56.780+0000 7f3736b84700 -1 bdev(0x565261220000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 13 2024-02-27T06:33:57.106 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:33:56 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[75036]: 2024-02-27T06:33:56.780+0000 7f373337d700 -1 bdev(0x565261220000 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 17 2024-02-27T06:33:57.111 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:33:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:33:56.779+0000 7f008ecf5700 -1 bdev(0x55955c8e2000 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 4 2024-02-27T06:33:58.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:57 smithi131 ceph-mon[66259]: pgmap v308: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 0 op/s 2024-02-27T06:33:58.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:57 smithi131 ceph-mon[66259]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-27T06:33:58.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:57 smithi131 ceph-mon[66259]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-27T06:33:58.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:57 smithi131 ceph-mon[66259]: osdmap e154: 8 total, 8 up, 8 in 2024-02-27T06:33:58.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:57 smithi131 ceph-mon[66259]: Standby daemon mds.foofs.smithi131.mhuwwb assigned to filesystem foofs as rank 0 2024-02-27T06:33:58.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:57 smithi131 ceph-mon[66259]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-27T06:33:58.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:57 smithi131 ceph-mon[66259]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-27T06:33:58.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:57 smithi131 ceph-mon[66259]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-02-27T06:33:58.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:57 smithi131 ceph-mon[66259]: fsmap foofs:1/1 {0=foofs.smithi131.mhuwwb=up:replay} 2024-02-27T06:33:58.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:33:58.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:57 smithi150 ceph-mon[52803]: pgmap v308: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 0 op/s 2024-02-27T06:33:58.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:57 smithi150 ceph-mon[52803]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-27T06:33:58.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:57 smithi150 ceph-mon[52803]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-27T06:33:58.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:57 smithi150 ceph-mon[52803]: osdmap e154: 8 total, 8 up, 8 in 2024-02-27T06:33:58.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:57 smithi150 ceph-mon[52803]: Standby daemon mds.foofs.smithi131.mhuwwb assigned to filesystem foofs as rank 0 2024-02-27T06:33:58.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:57 smithi150 ceph-mon[52803]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-27T06:33:58.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:57 smithi150 ceph-mon[52803]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-27T06:33:58.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:57 smithi150 ceph-mon[52803]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-02-27T06:33:58.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:57 smithi150 ceph-mon[52803]: fsmap foofs:1/1 {0=foofs.smithi131.mhuwwb=up:replay} 2024-02-27T06:33:58.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:33:59.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:58 smithi131 ceph-mon[66259]: reconnect by client.14538 172.21.15.131:0/2232015846 after 0.000999981 2024-02-27T06:33:59.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:58 smithi131 ceph-mon[66259]: reconnect by client.24335 172.21.15.150:0/1102554480 after 0.000999981 2024-02-27T06:33:59.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:58 smithi131 ceph-mon[66259]: mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640] up:reconnect 2024-02-27T06:33:59.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:58 smithi131 ceph-mon[66259]: fsmap foofs:1/1 {0=foofs.smithi131.mhuwwb=up:reconnect} 2024-02-27T06:33:59.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:58 smithi150 ceph-mon[52803]: reconnect by client.14538 172.21.15.131:0/2232015846 after 0.000999981 2024-02-27T06:33:59.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:58 smithi150 ceph-mon[52803]: reconnect by client.24335 172.21.15.150:0/1102554480 after 0.000999981 2024-02-27T06:33:59.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:58 smithi150 ceph-mon[52803]: mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640] up:reconnect 2024-02-27T06:33:59.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:58 smithi150 ceph-mon[52803]: fsmap foofs:1/1 {0=foofs.smithi131.mhuwwb=up:reconnect} 2024-02-27T06:34:00.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:59 smithi131 ceph-mon[66259]: pgmap v310: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 1 op/s 2024-02-27T06:34:00.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:59 smithi131 ceph-mon[66259]: mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640] up:rejoin 2024-02-27T06:34:00.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:59 smithi131 ceph-mon[66259]: fsmap foofs:1/1 {0=foofs.smithi131.mhuwwb=up:rejoin} 2024-02-27T06:34:00.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:33:59 smithi131 ceph-mon[66259]: daemon mds.foofs.smithi131.mhuwwb is now active in filesystem foofs as rank 0 2024-02-27T06:34:00.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:59 smithi150 ceph-mon[52803]: pgmap v310: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 1 op/s 2024-02-27T06:34:00.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:59 smithi150 ceph-mon[52803]: mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640] up:rejoin 2024-02-27T06:34:00.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:59 smithi150 ceph-mon[52803]: fsmap foofs:1/1 {0=foofs.smithi131.mhuwwb=up:rejoin} 2024-02-27T06:34:00.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:33:59 smithi150 ceph-mon[52803]: daemon mds.foofs.smithi131.mhuwwb is now active in filesystem foofs as rank 0 2024-02-27T06:34:01.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:00 smithi131 ceph-mon[66259]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-27T06:34:01.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:00 smithi131 ceph-mon[66259]: mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640] up:active 2024-02-27T06:34:01.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:00 smithi131 ceph-mon[66259]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 2024-02-27T06:34:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:00 smithi150 ceph-mon[52803]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-27T06:34:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:00 smithi150 ceph-mon[52803]: mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640] up:active 2024-02-27T06:34:01.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:00 smithi150 ceph-mon[52803]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 2024-02-27T06:34:02.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:01 smithi131 ceph-mon[66259]: pgmap v311: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 5 op/s 2024-02-27T06:34:02.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:01 smithi150 ceph-mon[52803]: pgmap v311: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 5 op/s 2024-02-27T06:34:04.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:03 smithi131 ceph-mon[66259]: pgmap v312: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 9 op/s 2024-02-27T06:34:04.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:03 smithi150 ceph-mon[52803]: pgmap v312: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 9 op/s 2024-02-27T06:34:06.045 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:05 smithi150 ceph-mon[52803]: pgmap v313: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 8.7 KiB/s wr, 14 op/s 2024-02-27T06:34:06.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:05 smithi131 ceph-mon[66259]: pgmap v313: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 8.7 KiB/s wr, 14 op/s 2024-02-27T06:34:07.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:07 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:07.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:07 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:07.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:07 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:34:07.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:07 smithi150 ceph-mon[52803]: pgmap v314: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 8.7 KiB/s wr, 14 op/s 2024-02-27T06:34:07.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:07 smithi150 ceph-mon[52803]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-27T06:34:07.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:07 smithi150 ceph-mon[52803]: Cluster is now healthy 2024-02-27T06:34:07.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:07 smithi150 ceph-mon[52803]: mds.? [v2:172.21.15.150:6832/1757949723,v1:172.21.15.150:6833/1757949723] up:boot 2024-02-27T06:34:07.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:07 smithi150 ceph-mon[52803]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 1 up:standby 2024-02-27T06:34:07.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:07 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:34:07.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:07 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:07.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:07 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:07.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:07 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:34:07.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:07 smithi131 ceph-mon[66259]: pgmap v314: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 8.7 KiB/s wr, 14 op/s 2024-02-27T06:34:07.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:07 smithi131 ceph-mon[66259]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-27T06:34:07.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:07 smithi131 ceph-mon[66259]: Cluster is now healthy 2024-02-27T06:34:07.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:07 smithi131 ceph-mon[66259]: mds.? [v2:172.21.15.150:6832/1757949723,v1:172.21.15.150:6833/1757949723] up:boot 2024-02-27T06:34:07.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:07 smithi131 ceph-mon[66259]: fsmap foofs:1 {0=foofs.smithi131.mhuwwb=up:active} 1 up:standby 2024-02-27T06:34:07.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:07 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:34:09.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:09 smithi131 ceph-mon[66259]: pgmap v315: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 7.4 KiB/s wr, 12 op/s 2024-02-27T06:34:09.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:09.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:09.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:09 smithi150 ceph-mon[52803]: pgmap v315: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 20 MiB/s rd, 7.4 KiB/s wr, 12 op/s 2024-02-27T06:34:09.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:09.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:11.257 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:34:11.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:11.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:11 smithi131 ceph-mon[66259]: pgmap v316: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 7.2 KiB/s wr, 11 op/s 2024-02-27T06:34:11.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:11.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:11 smithi131 ceph-mon[66259]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:11.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:34:11.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:11.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:11 smithi150 ceph-mon[52803]: pgmap v316: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s rd, 7.2 KiB/s wr, 11 op/s 2024-02-27T06:34:11.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:11.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:11 smithi150 ceph-mon[52803]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:11.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:34:11.914 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:34:11.914 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (10m) 23s ago 19m 22.5M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:34:11.914 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (6m) 23s ago 19m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:34:11.914 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (6m) 2s ago 18m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:34:11.914 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (9m) 23s ago 18m 52.6M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:34:11.914 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (29s) 23s ago 13m 14.7M - 19.0.0-1736-g635ee685 c4eb0759a73a 5b3c3fc0c3d9 2024-02-27T06:34:11.914 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (6s) 2s ago 13m 13.7M - 19.0.0-1736-g635ee685 c4eb0759a73a 2e0e5aff32ee 2024-02-27T06:34:11.914 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (9m) 23s ago 20m 512M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (8m) 2s ago 18m 452M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (7m) 23s ago 20m 58.5M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (7m) 2s ago 18m 46.9M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (13m) 23s ago 13m 280M - 4.0 e1d6a67b021e fed1febd7b1d 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (12m) 2s ago 12m 84.0M - 4.0 e1d6a67b021e fbd32032c95a 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (9m) 23s ago 18m 19.7M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (9m) 2s ago 18m 17.4M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (5m) 23s ago 17m 417M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (4m) 23s ago 17m 374M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (4m) 23s ago 16m 370M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (3m) 23s ago 16m 426M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 3d240baa2aed 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (2m) 2s ago 16m 354M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 7dc53922c309 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (2m) 2s ago 15m 323M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 6865822553f8 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (95s) 2s ago 15m 308M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 1dedb9e4ac79 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (57s) 2s ago 14m 317M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 50595e9edf32 2024-02-27T06:34:11.915 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (8m) 23s ago 18m 68.2M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:34:12.318 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 14 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:34:12.319 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:34:12.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:12 smithi131 ceph-mon[66259]: from='client.34498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:12.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:12 smithi131 ceph-mon[66259]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:12.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:12 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/2357701722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:12.639 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:12 smithi150 ceph-mon[52803]: from='client.34498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:12.639 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:12 smithi150 ceph-mon[52803]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:12.639 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:12 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/2357701722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:12.657 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:34:12.657 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout: "mds", 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout: "osd", 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "16/23 daemons upgraded", 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading mds daemons", 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:34:12.658 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:34:13.057 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:34:13.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:13 smithi131 ceph-mon[66259]: pgmap v317: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-02-27T06:34:13.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:13 smithi131 ceph-mon[66259]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:13.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:13 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/2494071343' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:34:13.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:13 smithi150 ceph-mon[52803]: pgmap v317: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.0 MiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-02-27T06:34:13.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:13 smithi150 ceph-mon[52803]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:13.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:13 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/2494071343' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:34:15.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: Detected new or changed devices on smithi150 2024-02-27T06:34:15.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:34:15.076 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: Upgrade: Setting container_image for all mds 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi131.mhuwwb"}]': finished 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi150.vsnjhu"}]': finished 2024-02-27T06:34:15.077 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.078 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: Upgrade: Setting container_image for all rgw 2024-02-27T06:34:15.078 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.078 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.078 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: Upgrade: Setting container_image for all rbd-mirror 2024-02-27T06:34:15.078 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.078 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.078 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.078 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: Upgrade: Setting container_image for all ceph-exporter 2024-02-27T06:34:15.078 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.078 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.078 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: Upgrade: Setting container_image for all iscsi 2024-02-27T06:34:15.078 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:14 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: Detected new or changed devices on smithi150 2024-02-27T06:34:15.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:34:15.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:34:15.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:34:15.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: Upgrade: Setting container_image for all mds 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi131.mhuwwb"}]: dispatch 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi131.mhuwwb"}]': finished 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi150.vsnjhu"}]: dispatch 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi150.vsnjhu"}]': finished 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: Upgrade: Setting container_image for all rgw 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: Upgrade: Setting container_image for all rbd-mirror 2024-02-27T06:34:15.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.364 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.364 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.364 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: Upgrade: Setting container_image for all ceph-exporter 2024-02-27T06:34:15.364 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:15.364 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:15.364 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: Upgrade: Setting container_image for all iscsi 2024-02-27T06:34:15.364 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:14 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:16.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:16 smithi131 ceph-mon[66259]: pgmap v318: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.4 KiB/s wr, 4 op/s 2024-02-27T06:34:16.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:16 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:16.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:16 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi131.cezbzx", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-27T06:34:16.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:16 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-27T06:34:16.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:16 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-27T06:34:16.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:16 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:34:16.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:16 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-27T06:34:16.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:16 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-27T06:34:16.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:16 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi131.cezbzx-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T06:34:16.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:16 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:34:16.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:16 smithi150 ceph-mon[52803]: pgmap v318: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.4 KiB/s wr, 4 op/s 2024-02-27T06:34:16.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:16 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:16.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:16 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi131.cezbzx", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-27T06:34:16.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:16 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-27T06:34:16.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:16 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-27T06:34:16.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:16 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:34:16.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:16 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-27T06:34:16.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:16 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-27T06:34:16.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:16 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi131.cezbzx-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T06:34:16.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:16 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:34:17.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:17 smithi131 ceph-mon[66259]: Upgrade: Updating nfs.foo.0.0.smithi131.cezbzx (1/2) 2024-02-27T06:34:17.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:17 smithi131 ceph-mon[66259]: Creating key for client.nfs.foo.0.0.smithi131.cezbzx 2024-02-27T06:34:17.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:17 smithi131 ceph-mon[66259]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-02-27T06:34:17.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:17 smithi131 ceph-mon[66259]: Rados config object exists: conf-nfs.foo 2024-02-27T06:34:17.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:17 smithi131 ceph-mon[66259]: Creating key for client.nfs.foo.0.0.smithi131.cezbzx-rgw 2024-02-27T06:34:17.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:17 smithi131 ceph-mon[66259]: Bind address in nfs.foo.0.0.smithi131.cezbzx's ganesha conf is defaulting to empty 2024-02-27T06:34:17.607 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:17 smithi131 ceph-mon[66259]: Deploying daemon nfs.foo.0.0.smithi131.cezbzx on smithi131 2024-02-27T06:34:17.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:17 smithi150 ceph-mon[52803]: Upgrade: Updating nfs.foo.0.0.smithi131.cezbzx (1/2) 2024-02-27T06:34:17.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:17 smithi150 ceph-mon[52803]: Creating key for client.nfs.foo.0.0.smithi131.cezbzx 2024-02-27T06:34:17.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:17 smithi150 ceph-mon[52803]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-02-27T06:34:17.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:17 smithi150 ceph-mon[52803]: Rados config object exists: conf-nfs.foo 2024-02-27T06:34:17.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:17 smithi150 ceph-mon[52803]: Creating key for client.nfs.foo.0.0.smithi131.cezbzx-rgw 2024-02-27T06:34:17.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:17 smithi150 ceph-mon[52803]: Bind address in nfs.foo.0.0.smithi131.cezbzx's ganesha conf is defaulting to empty 2024-02-27T06:34:17.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:17 smithi150 ceph-mon[52803]: Deploying daemon nfs.foo.0.0.smithi131.cezbzx on smithi131 2024-02-27T06:34:18.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:18 smithi131 ceph-mon[66259]: pgmap v319: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 426 B/s rd, 170 B/s wr, 0 op/s 2024-02-27T06:34:18.513 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:18 smithi150 ceph-mon[52803]: pgmap v319: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 426 B/s rd, 170 B/s wr, 0 op/s 2024-02-27T06:34:20.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:20 smithi131 ceph-mon[66259]: pgmap v320: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 170 B/s wr, 0 op/s 2024-02-27T06:34:20.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:20 smithi131 ceph-mon[66259]: from='mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640]' entity='mds.foofs.smithi131.mhuwwb' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.131:0/2232015846"}]: dispatch 2024-02-27T06:34:20.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:20.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:20.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:20 smithi150 ceph-mon[52803]: pgmap v320: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 170 B/s wr, 0 op/s 2024-02-27T06:34:20.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:20 smithi150 ceph-mon[52803]: from='mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640]' entity='mds.foofs.smithi131.mhuwwb' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.131:0/2232015846"}]: dispatch 2024-02-27T06:34:20.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:20.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:21.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:21 smithi150 ceph-mon[52803]: Evicting (and blocklisting) client session 14538 (172.21.15.131:0/2232015846) 2024-02-27T06:34:21.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:21 smithi150 ceph-mon[52803]: from='mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640]' entity='mds.foofs.smithi131.mhuwwb' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.131:0/2232015846"}]': finished 2024-02-27T06:34:21.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:21 smithi150 ceph-mon[52803]: osdmap e155: 8 total, 8 up, 8 in 2024-02-27T06:34:21.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:21 smithi131 ceph-mon[66259]: Evicting (and blocklisting) client session 14538 (172.21.15.131:0/2232015846) 2024-02-27T06:34:21.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:21 smithi131 ceph-mon[66259]: from='mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640]' entity='mds.foofs.smithi131.mhuwwb' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.131:0/2232015846"}]': finished 2024-02-27T06:34:21.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:21 smithi131 ceph-mon[66259]: osdmap e155: 8 total, 8 up, 8 in 2024-02-27T06:34:22.480 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:22 smithi150 ceph-mon[52803]: pgmap v322: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 307 B/s rd, 307 B/s wr, 0 op/s 2024-02-27T06:34:22.480 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:22.480 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi150.xgkkez", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-27T06:34:22.480 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-27T06:34:22.480 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-27T06:34:22.481 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:34:22.481 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-27T06:34:22.481 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-27T06:34:22.481 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi150.xgkkez-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T06:34:22.481 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:22 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:34:22.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:22 smithi131 ceph-mon[66259]: pgmap v322: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 307 B/s rd, 307 B/s wr, 0 op/s 2024-02-27T06:34:22.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:22.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi150.xgkkez", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-02-27T06:34:22.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-02-27T06:34:22.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-02-27T06:34:22.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:34:22.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-02-27T06:34:22.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-02-27T06:34:22.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi150.xgkkez-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-27T06:34:22.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:22 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:34:23.249 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:23 smithi150 ceph-mon[52803]: Upgrade: Updating nfs.foo.1.0.smithi150.xgkkez (2/2) 2024-02-27T06:34:23.250 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:23 smithi150 ceph-mon[52803]: Creating key for client.nfs.foo.1.0.smithi150.xgkkez 2024-02-27T06:34:23.250 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:23 smithi150 ceph-mon[52803]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-02-27T06:34:23.250 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:23 smithi150 ceph-mon[52803]: Rados config object exists: conf-nfs.foo 2024-02-27T06:34:23.250 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:23 smithi150 ceph-mon[52803]: Creating key for client.nfs.foo.1.0.smithi150.xgkkez-rgw 2024-02-27T06:34:23.250 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:23 smithi150 ceph-mon[52803]: Bind address in nfs.foo.1.0.smithi150.xgkkez's ganesha conf is defaulting to empty 2024-02-27T06:34:23.250 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:23 smithi150 ceph-mon[52803]: Deploying daemon nfs.foo.1.0.smithi150.xgkkez on smithi150 2024-02-27T06:34:23.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:23 smithi131 ceph-mon[66259]: Upgrade: Updating nfs.foo.1.0.smithi150.xgkkez (2/2) 2024-02-27T06:34:23.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:23 smithi131 ceph-mon[66259]: Creating key for client.nfs.foo.1.0.smithi150.xgkkez 2024-02-27T06:34:23.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:23 smithi131 ceph-mon[66259]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-02-27T06:34:23.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:23 smithi131 ceph-mon[66259]: Rados config object exists: conf-nfs.foo 2024-02-27T06:34:23.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:23 smithi131 ceph-mon[66259]: Creating key for client.nfs.foo.1.0.smithi150.xgkkez-rgw 2024-02-27T06:34:23.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:23 smithi131 ceph-mon[66259]: Bind address in nfs.foo.1.0.smithi150.xgkkez's ganesha conf is defaulting to empty 2024-02-27T06:34:23.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:23 smithi131 ceph-mon[66259]: Deploying daemon nfs.foo.1.0.smithi150.xgkkez on smithi150 2024-02-27T06:34:24.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:24 smithi150 ceph-mon[52803]: pgmap v323: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 819 B/s wr, 3 op/s 2024-02-27T06:34:24.607 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:24 smithi131 ceph-mon[66259]: pgmap v323: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 819 B/s wr, 3 op/s 2024-02-27T06:34:26.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:26 smithi131 ceph-mon[66259]: pgmap v324: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-27T06:34:26.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:26.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:26.449 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:34:26.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:26 smithi150 ceph-mon[52803]: pgmap v324: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-27T06:34:26.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:26.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:26.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:34:27.258 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:34:27.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:34:28.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:28 smithi131 ceph-mon[66259]: pgmap v325: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-27T06:34:28.611 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:28 smithi150 ceph-mon[52803]: pgmap v325: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-27T06:34:29.618 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:29 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:29.619 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:29 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:29.619 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:29 smithi131 ceph-mon[66259]: Evicting (and blocklisting) client session 24335 (172.21.15.150:0/1102554480) 2024-02-27T06:34:29.619 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:29 smithi131 ceph-mon[66259]: from='mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640]' entity='mds.foofs.smithi131.mhuwwb' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/1102554480"}]: dispatch 2024-02-27T06:34:29.619 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:29 smithi131 ceph-mon[66259]: pgmap v326: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-27T06:34:29.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:29 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:29.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:29 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:29.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:29 smithi150 ceph-mon[52803]: Evicting (and blocklisting) client session 24335 (172.21.15.150:0/1102554480) 2024-02-27T06:34:29.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:29 smithi150 ceph-mon[52803]: from='mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640]' entity='mds.foofs.smithi131.mhuwwb' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/1102554480"}]: dispatch 2024-02-27T06:34:29.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:29 smithi150 ceph-mon[52803]: pgmap v326: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-27T06:34:30.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:30 smithi150 ceph-mon[52803]: from='mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640]' entity='mds.foofs.smithi131.mhuwwb' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/1102554480"}]': finished 2024-02-27T06:34:30.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:30 smithi150 ceph-mon[52803]: osdmap e156: 8 total, 8 up, 8 in 2024-02-27T06:34:30.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:30.612 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:30 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:30.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:30 smithi131 ceph-mon[66259]: from='mds.? [v2:172.21.15.131:6834/2809108640,v1:172.21.15.131:6835/2809108640]' entity='mds.foofs.smithi131.mhuwwb' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.150:0/1102554480"}]': finished 2024-02-27T06:34:30.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:30 smithi131 ceph-mon[66259]: osdmap e156: 8 total, 8 up, 8 in 2024-02-27T06:34:30.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:30.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:30 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:32.322 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:31 smithi150 ceph-mon[52803]: pgmap v328: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-02-27T06:34:32.322 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:32.322 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:31 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:32.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:31 smithi131 ceph-mon[66259]: pgmap v328: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-02-27T06:34:32.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:32.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:31 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:33.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:32 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:33.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:32 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:33.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:32 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:33.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:32 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:33.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:32 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:33.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:32 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:33.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:32 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:33.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:32 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:34.222 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:34 smithi131 ceph-mon[66259]: pgmap v329: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-02-27T06:34:34.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:34 smithi150 ceph-mon[52803]: pgmap v329: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-02-27T06:34:36.333 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: pgmap v330: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-27T06:34:36.333 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:36.333 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.334 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.335 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:36.335 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi131.cezbzx"}]: dispatch 2024-02-27T06:34:36.335 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi131.cezbzx"}]': finished 2024-02-27T06:34:36.335 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi150.xgkkez"}]: dispatch 2024-02-27T06:34:36.335 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi150.xgkkez"}]': finished 2024-02-27T06:34:36.335 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.335 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:36 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:36.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: pgmap v330: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-27T06:34:36.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:36.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:36.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:34:36.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:34:36.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:36.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:36.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:34:36.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi131.cezbzx"}]: dispatch 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi131.cezbzx"}]': finished 2024-02-27T06:34:36.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi150.xgkkez"}]: dispatch 2024-02-27T06:34:36.364 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi150.xgkkez"}]': finished 2024-02-27T06:34:36.364 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:36.364 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:36 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:37.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:37 smithi131 ceph-mon[66259]: Upgrade: Setting container_image for all nfs 2024-02-27T06:34:37.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:37 smithi131 ceph-mon[66259]: Upgrade: Setting container_image for all nvmeof 2024-02-27T06:34:37.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:37 smithi150 ceph-mon[52803]: Upgrade: Setting container_image for all nfs 2024-02-27T06:34:37.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:37 smithi150 ceph-mon[52803]: Upgrade: Setting container_image for all nvmeof 2024-02-27T06:34:38.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:38 smithi131 ceph-mon[66259]: pgmap v331: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 409 B/s wr, 0 op/s 2024-02-27T06:34:38.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:38 smithi150 ceph-mon[52803]: pgmap v331: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 409 B/s wr, 0 op/s 2024-02-27T06:34:39.281 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:39 smithi131 ceph-mon[66259]: Upgrade: Updating node-exporter.smithi131 (1/2) 2024-02-27T06:34:39.282 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:39 smithi131 ceph-mon[66259]: Deploying daemon node-exporter.smithi131 on smithi131 2024-02-27T06:34:39.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:39 smithi150 ceph-mon[52803]: Upgrade: Updating node-exporter.smithi131 (1/2) 2024-02-27T06:34:39.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:39 smithi150 ceph-mon[52803]: Deploying daemon node-exporter.smithi131 on smithi131 2024-02-27T06:34:40.287 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:40 smithi131 ceph-mon[66259]: pgmap v332: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 409 B/s wr, 0 op/s 2024-02-27T06:34:40.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:40 smithi150 ceph-mon[52803]: pgmap v332: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 102 B/s rd, 409 B/s wr, 0 op/s 2024-02-27T06:34:42.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:42 smithi131 ceph-mon[66259]: pgmap v333: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 91 B/s rd, 366 B/s wr, 0 op/s 2024-02-27T06:34:42.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:42 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:34:42.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:42 smithi150 ceph-mon[52803]: pgmap v333: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 91 B/s rd, 366 B/s wr, 0 op/s 2024-02-27T06:34:42.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:42 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:34:43.412 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:34:44.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:44 smithi131 ceph-mon[66259]: pgmap v334: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 341 B/s wr, 0 op/s 2024-02-27T06:34:44.161 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:34:44.161 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (10m) 13s ago 19m 22.6M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (6m) 13s ago 19m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (6m) 15s ago 19m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (10m) 13s ago 19m 53.1M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (61s) 13s ago 13m 35.7M - 19.0.0-1736-g635ee685 c4eb0759a73a 5b3c3fc0c3d9 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (38s) 15s ago 13m 13.8M - 19.0.0-1736-g635ee685 c4eb0759a73a 2e0e5aff32ee 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (9m) 13s ago 20m 513M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (8m) 15s ago 19m 453M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (8m) 13s ago 21m 60.5M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (7m) 15s ago 19m 47.3M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (24s) 13s ago 13m 68.8M - 5.7 c4eb0759a73a 5340eb3423e8 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (19s) 15s ago 13m 17.9M - 5.7 c4eb0759a73a 75f5b5dbf7dc 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (10m) 13s ago 19m 17.7M - 1.3.1 1dbe0e931976 ec6e317b931e 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (9m) 15s ago 18m 20.2M - 1.3.1 1dbe0e931976 f341a0513b92 2024-02-27T06:34:44.162 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (6m) 13s ago 18m 419M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:34:44.163 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (5m) 13s ago 17m 376M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:34:44.163 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (4m) 13s ago 17m 372M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:34:44.163 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (4m) 13s ago 16m 429M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 3d240baa2aed 2024-02-27T06:34:44.163 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (3m) 15s ago 16m 354M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 7dc53922c309 2024-02-27T06:34:44.163 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (2m) 15s ago 16m 324M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 6865822553f8 2024-02-27T06:34:44.163 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (2m) 15s ago 15m 307M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 1dedb9e4ac79 2024-02-27T06:34:44.163 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (90s) 15s ago 15m 317M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 50595e9edf32 2024-02-27T06:34:44.163 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (8m) 13s ago 18m 68.2M - 2.33.4 514e6a882f6e b7bd8c104956 2024-02-27T06:34:44.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:44 smithi150 ceph-mon[52803]: pgmap v334: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 341 B/s wr, 0 op/s 2024-02-27T06:34:44.582 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:34:44.582 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:34:44.582 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:34:44.582 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 14 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:34:44.583 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:34:45.013 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:34:45.013 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:34:45.013 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:34:45.013 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:34:45.013 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:34:45.013 INFO:teuthology.orchestra.run.smithi131.stdout: "mds", 2024-02-27T06:34:45.013 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:34:45.013 INFO:teuthology.orchestra.run.smithi131.stdout: "osd", 2024-02-27T06:34:45.013 INFO:teuthology.orchestra.run.smithi131.stdout: "nfs", 2024-02-27T06:34:45.013 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:34:45.013 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:34:45.014 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:34:45.014 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "18/23 daemons upgraded", 2024-02-27T06:34:45.014 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading node-exporter daemons", 2024-02-27T06:34:45.014 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:34:45.014 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:34:45.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:45 smithi131 ceph-mon[66259]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:45.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:45 smithi131 ceph-mon[66259]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:45.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:45 smithi131 ceph-mon[66259]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:45.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:45 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/1138143287' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:45.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:45 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:45.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:45 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:45.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:45 smithi150 ceph-mon[52803]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:45.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:45 smithi150 ceph-mon[52803]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:45.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:45 smithi150 ceph-mon[52803]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:45.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:45 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/1138143287' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:34:45.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:45 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:45.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:45 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:45.409 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:34:46.327 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:46 smithi150 ceph-mon[52803]: pgmap v335: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-02-27T06:34:46.327 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:46 smithi150 ceph-mon[52803]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:46.327 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:46 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/53847202' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:34:46.331 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:46 smithi131 ceph-mon[66259]: pgmap v335: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-02-27T06:34:46.331 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:46 smithi131 ceph-mon[66259]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:34:46.331 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:46 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/53847202' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:34:48.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:48 smithi150 ceph-mon[52803]: Upgrade: Updating node-exporter.smithi150 (2/2) 2024-02-27T06:34:48.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:48 smithi150 ceph-mon[52803]: Deploying daemon node-exporter.smithi150 on smithi150 2024-02-27T06:34:48.309 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:48 smithi150 ceph-mon[52803]: pgmap v336: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:34:48.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:48 smithi131 ceph-mon[66259]: Upgrade: Updating node-exporter.smithi150 (2/2) 2024-02-27T06:34:48.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:48 smithi131 ceph-mon[66259]: Deploying daemon node-exporter.smithi150 on smithi150 2024-02-27T06:34:48.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:48 smithi131 ceph-mon[66259]: pgmap v336: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:34:50.329 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:50 smithi150 ceph-mon[52803]: pgmap v337: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:34:50.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:50 smithi131 ceph-mon[66259]: pgmap v337: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:34:52.327 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:52 smithi131 ceph-mon[66259]: pgmap v338: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:34:52.328 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:52 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:52.328 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:52 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:52.328 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:52 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:34:52.328 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:52 smithi150 ceph-mon[52803]: pgmap v338: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:34:52.328 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:52 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:52.328 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:52 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:52.328 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:52 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:34:54.303 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:54 smithi150 ceph-mon[52803]: pgmap v339: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:34:54.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:54 smithi131 ceph-mon[66259]: pgmap v339: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:34:55.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:55 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:55.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:55 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:55.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:55 smithi131 ceph-mon[66259]: pgmap v340: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:34:55.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:55 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:55.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:55 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:55.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:55 smithi150 ceph-mon[52803]: pgmap v340: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:34:56.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:56 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:56.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:56 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:56.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:56 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:34:56.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:56 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:56.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:56 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:56.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:56 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:34:57.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:57 smithi131 ceph-mon[66259]: pgmap v341: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:34:57.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:57.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:57.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:57 smithi150 ceph-mon[52803]: pgmap v341: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:34:57.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:57.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:58.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:58.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:58 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:59.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:34:59.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:58 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:00.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:59 smithi131 ceph-mon[66259]: pgmap v342: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:00.126 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:59 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:00.126 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:34:59 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:00.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:59 smithi150 ceph-mon[52803]: pgmap v342: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:00.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:59 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:00.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:34:59 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:02.315 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: pgmap v343: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:02.315 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:02.315 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:02.315 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:35:02.315 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:35:02.315 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:02.315 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:02.315 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:01 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: pgmap v343: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:02.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:02.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:02.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:35:02.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:35:02.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:02.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:02.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:02.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:02.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:01 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:04.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:03 smithi131 ceph-mon[66259]: pgmap v344: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:35:04.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:03 smithi150 ceph-mon[52803]: pgmap v344: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:35:05.201 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:04 smithi131 ceph-mon[66259]: Upgrade: Updating prometheus.smithi131 2024-02-27T06:35:05.201 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:04 smithi131 ceph-mon[66259]: Deploying daemon prometheus.smithi131 on smithi131 2024-02-27T06:35:05.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:04 smithi150 ceph-mon[52803]: Upgrade: Updating prometheus.smithi131 2024-02-27T06:35:05.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:04 smithi150 ceph-mon[52803]: Deploying daemon prometheus.smithi131 on smithi131 2024-02-27T06:35:06.332 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:05 smithi131 ceph-mon[66259]: pgmap v345: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:06.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:05 smithi150 ceph-mon[52803]: pgmap v345: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:08.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:07 smithi131 ceph-mon[66259]: pgmap v346: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:08.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:07 smithi150 ceph-mon[52803]: pgmap v346: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:10.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:09 smithi131 ceph-mon[66259]: pgmap v347: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:10.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:10.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:10.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:10.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:10.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:09 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:10.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:09 smithi150 ceph-mon[52803]: pgmap v347: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:10.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:10.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:10.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:10.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:10.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:09 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:12.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:11 smithi131 ceph-mon[66259]: pgmap v348: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:12.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:35:12.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:11 smithi150 ceph-mon[52803]: pgmap v348: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:12.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:35:14.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:13 smithi131 ceph-mon[66259]: pgmap v349: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:35:14.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:13 smithi150 ceph-mon[52803]: pgmap v349: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:35:15.775 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:35:15.775 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:15 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:15.776 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:15 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:15.776 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:15 smithi131 ceph-mon[66259]: pgmap v350: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:15.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:15 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:15.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:15 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:15.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:15 smithi150 ceph-mon[52803]: pgmap v350: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:16.455 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:35:16.455 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (11m) 2s ago 20m 22.7M - 0.23.0 ba2b418f427c be99c31e146a 2024-02-27T06:35:16.455 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (7m) 2s ago 20m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:35:16.455 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (7m) 21s ago 19m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:35:16.455 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (10m) 2s ago 20m 53.0M - 8.3.5 dad864ee21e9 644c49d00440 2024-02-27T06:35:16.455 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (94s) 2s ago 14m 36.1M - 19.0.0-1736-g635ee685 c4eb0759a73a 5b3c3fc0c3d9 2024-02-27T06:35:16.455 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (70s) 21s ago 14m 13.9M - 19.0.0-1736-g635ee685 c4eb0759a73a 2e0e5aff32ee 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (10m) 2s ago 21m 516M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (9m) 21s ago 19m 453M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (9m) 2s ago 21m 61.7M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (8m) 21s ago 19m 48.6M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (56s) 2s ago 14m 69.1M - 5.7 c4eb0759a73a 5340eb3423e8 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (51s) 21s ago 14m 68.5M - 5.7 c4eb0759a73a 75f5b5dbf7dc 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (32s) 2s ago 19m 8308k - 1.5.0 0da6a335fe13 bf1053818375 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (25s) 21s ago 19m 3581k - 1.5.0 0da6a335fe13 e1c6d1fee8ba 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (6m) 2s ago 18m 418M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (6m) 2s ago 18m 376M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (5m) 2s ago 17m 374M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (4m) 2s ago 17m 429M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 3d240baa2aed 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (3m) 21s ago 17m 355M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 7dc53922c309 2024-02-27T06:35:16.456 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (3m) 21s ago 16m 324M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 6865822553f8 2024-02-27T06:35:16.457 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (2m) 21s ago 16m 307M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 1dedb9e4ac79 2024-02-27T06:35:16.457 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (2m) 21s ago 15m 317M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 50595e9edf32 2024-02-27T06:35:16.457 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (8s) 2s ago 19m 42.3M - 2.43.0 a07b618ecd1d c7dcd7362c89 2024-02-27T06:35:16.605 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:16 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:16.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:16 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:16.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:16 smithi131 ceph-mon[66259]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:16.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:16 smithi131 ceph-mon[66259]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:16.851 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:35:16.851 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:35:16.851 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:35:16.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:35:16.851 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:35:16.851 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:35:16.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:35:16.851 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:35:16.851 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T06:35:16.851 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:35:16.851 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:35:16.851 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:35:16.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:35:16.852 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:35:16.852 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 14 2024-02-27T06:35:16.852 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:35:16.852 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:35:16.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:16 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:16.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:16 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:16.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:16 smithi150 ceph-mon[52803]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:16.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:16 smithi150 ceph-mon[52803]: from='client.34582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:17.194 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "mds", 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "osd", 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "nfs", 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "18/23 daemons upgraded", 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading prometheus daemons", 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:35:17.195 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:35:17.511 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:17 smithi131 ceph-mon[66259]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:17.511 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:17 smithi131 ceph-mon[66259]: pgmap v351: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:17.511 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:17 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/915102191' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:17.511 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:17 smithi131 ceph-mon[66259]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:17.604 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:35:17.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:17 smithi150 ceph-mon[52803]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:17.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:17 smithi150 ceph-mon[52803]: pgmap v351: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:17.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:17 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/915102191' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:17.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:17 smithi150 ceph-mon[52803]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:18.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:18 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/227172806' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:35:18.763 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:18 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/227172806' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:35:20.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: pgmap v352: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:20.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.357 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.358 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.358 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.358 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.358 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.358 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:20 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: pgmap v352: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:20.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:20.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.362 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:20.363 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:20 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:22.225 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:22 smithi131 ceph-mon[66259]: pgmap v353: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:22.225 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:22 smithi131 ceph-mon[66259]: Upgrade: Updating alertmanager.smithi131 2024-02-27T06:35:22.225 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:22 smithi131 ceph-mon[66259]: Deploying daemon alertmanager.smithi131 on smithi131 2024-02-27T06:35:22.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:22 smithi150 ceph-mon[52803]: pgmap v353: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:22.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:22 smithi150 ceph-mon[52803]: Upgrade: Updating alertmanager.smithi131 2024-02-27T06:35:22.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:22 smithi150 ceph-mon[52803]: Deploying daemon alertmanager.smithi131 on smithi131 2024-02-27T06:35:24.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:24 smithi131 ceph-mon[66259]: pgmap v354: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:35:24.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:24 smithi150 ceph-mon[52803]: pgmap v354: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:35:26.211 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:26 smithi131 ceph-mon[66259]: pgmap v355: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:26.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:26 smithi150 ceph-mon[52803]: pgmap v355: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:27.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:27.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:27.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:35:27.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:27 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:27.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:27 smithi131 ceph-mon[66259]: pgmap v356: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-27T06:35:27.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:27.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:27.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:35:27.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:27 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:27.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:27 smithi150 ceph-mon[52803]: pgmap v356: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-27T06:35:30.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:29 smithi131 ceph-mon[66259]: pgmap v357: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-27T06:35:30.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:29 smithi150 ceph-mon[52803]: pgmap v357: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-27T06:35:32.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:31 smithi131 ceph-mon[66259]: pgmap v358: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-27T06:35:32.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:31 smithi150 ceph-mon[52803]: pgmap v358: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-27T06:35:33.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:32 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:33.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:32 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:33.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:32 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:33.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:32 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:34.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:33 smithi131 ceph-mon[66259]: pgmap v359: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 335 B/s rd, 0 op/s 2024-02-27T06:35:34.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:33 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:34.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:33 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:34.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:33 smithi150 ceph-mon[52803]: pgmap v359: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 335 B/s rd, 0 op/s 2024-02-27T06:35:34.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:33 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:34.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:33 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:36.316 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:35 smithi131 ceph-mon[66259]: pgmap v360: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-27T06:35:36.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:35 smithi150 ceph-mon[52803]: pgmap v360: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-27T06:35:37.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.857 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:37 smithi131 ceph-mon[66259]: pgmap v361: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-27T06:35:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:35:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:35:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T06:35:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-27T06:35:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:37.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.862 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:37.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:37 smithi150 ceph-mon[52803]: pgmap v361: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-27T06:35:38.717 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:38 smithi131 ceph-mon[66259]: Upgrade: Updating grafana.smithi131 2024-02-27T06:35:38.717 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:38 smithi131 ceph-mon[66259]: Deploying daemon grafana.smithi131 on smithi131 2024-02-27T06:35:38.863 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:38 smithi150 ceph-mon[52803]: Upgrade: Updating grafana.smithi131 2024-02-27T06:35:38.864 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:38 smithi150 ceph-mon[52803]: Deploying daemon grafana.smithi131 on smithi131 2024-02-27T06:35:39.606 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:39 smithi131 ceph-mon[66259]: pgmap v362: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:39.861 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:39 smithi150 ceph-mon[52803]: pgmap v362: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:42.086 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:41 smithi131 ceph-mon[66259]: pgmap v363: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:42.086 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:35:42.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:41 smithi150 ceph-mon[52803]: pgmap v363: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:42.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:35:43.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:43 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:43.855 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:43 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:43.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:43 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:43.856 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:43 smithi131 ceph-mon[66259]: pgmap v364: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:35:44.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:43 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:44.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:43 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:44.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:43 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:44.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:43 smithi150 ceph-mon[52803]: pgmap v364: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:35:46.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:45 smithi131 ceph-mon[66259]: pgmap v365: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:46.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:45 smithi150 ceph-mon[52803]: pgmap v365: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:47.964 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:35:48.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:47 smithi131 ceph-mon[66259]: pgmap v366: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:48.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:47 smithi150 ceph-mon[52803]: pgmap v366: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (22s) 0s ago 20m 14.9M - 0.25.0 c8568f914cd2 d21da279d6aa 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (8m) 0s ago 20m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (7m) 54s ago 20m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (6s) 0s ago 20m 51.6M - 9.4.12 a0ea5e3c46d9 3286cc15d984 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (2m) 0s ago 14m 36.4M - 19.0.0-1736-g635ee685 c4eb0759a73a 5b3c3fc0c3d9 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (102s) 54s ago 14m 13.9M - 19.0.0-1736-g635ee685 c4eb0759a73a 2e0e5aff32ee 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (10m) 0s ago 21m 516M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (9m) 54s ago 20m 453M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (9m) 0s ago 22m 63.0M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (9m) 54s ago 20m 48.6M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (89s) 0s ago 14m 69.1M - 5.7 c4eb0759a73a 5340eb3423e8 2024-02-27T06:35:48.704 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (83s) 54s ago 14m 68.5M - 5.7 c4eb0759a73a 75f5b5dbf7dc 2024-02-27T06:35:48.705 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (64s) 0s ago 20m 8799k - 1.5.0 0da6a335fe13 bf1053818375 2024-02-27T06:35:48.705 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (57s) 54s ago 19m 3581k - 1.5.0 0da6a335fe13 e1c6d1fee8ba 2024-02-27T06:35:48.705 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (7m) 0s ago 19m 420M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:35:48.705 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (6m) 0s ago 18m 377M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:35:48.705 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (5m) 0s ago 18m 374M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:35:48.705 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (5m) 0s ago 17m 430M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 3d240baa2aed 2024-02-27T06:35:48.705 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (4m) 54s ago 17m 355M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 7dc53922c309 2024-02-27T06:35:48.705 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (3m) 54s ago 17m 324M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 6865822553f8 2024-02-27T06:35:48.705 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (3m) 54s ago 16m 307M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 1dedb9e4ac79 2024-02-27T06:35:48.705 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (2m) 54s ago 16m 317M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 50595e9edf32 2024-02-27T06:35:48.705 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (40s) 0s ago 19m 51.5M - 2.43.0 a07b618ecd1d c7dcd7362c89 2024-02-27T06:35:49.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:49.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:48 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:49.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:48 smithi131 ceph-mon[66259]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:49.115 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:35:49.115 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:35:49.115 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:35:49.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:35:49.115 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:35:49.115 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:35:49.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:35:49.115 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:35:49.115 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T06:35:49.115 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:35:49.116 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:35:49.116 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:35:49.116 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:35:49.116 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:35:49.116 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 14 2024-02-27T06:35:49.116 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:35:49.116 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:35:49.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:49.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:48 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:49.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:48 smithi150 ceph-mon[52803]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:49.456 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:35:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", 2024-02-27T06:35:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": true, 2024-02-27T06:35:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-27T06:35:49.456 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [ 2024-02-27T06:35:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "mds", 2024-02-27T06:35:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "mon", 2024-02-27T06:35:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "osd", 2024-02-27T06:35:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "nfs", 2024-02-27T06:35:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr", 2024-02-27T06:35:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "crash" 2024-02-27T06:35:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:35:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": "18/23 daemons upgraded", 2024-02-27T06:35:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "Currently upgrading grafana daemons", 2024-02-27T06:35:49.457 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:35:49.457 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:35:49.866 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:35:50.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:50 smithi131 ceph-mon[66259]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:50.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:50 smithi131 ceph-mon[66259]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:50.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:50 smithi131 ceph-mon[66259]: pgmap v367: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:50.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:50 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:50.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:50 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:50.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:50 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/1408430445' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:50.356 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:50 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/3633269376' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:35:50.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:50 smithi150 ceph-mon[52803]: from='client.34606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:50.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:50 smithi150 ceph-mon[52803]: from='client.34610 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:50.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:50 smithi150 ceph-mon[52803]: pgmap v367: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:50.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:50 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:50.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:50 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:50.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:50 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/1408430445' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:50.361 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:50 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/3633269376' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:35:51.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:51 smithi131 ceph-mon[66259]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:51.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:51 smithi150 ceph-mon[52803]: from='client.34618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:35:52.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:52 smithi131 ceph-mon[66259]: pgmap v368: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:52.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:52 smithi150 ceph-mon[52803]: pgmap v368: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.106 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: pgmap v369: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.107 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: Upgrade: Finalizing container_image settings 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-27T06:35:54.108 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.109 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: Upgrade: Complete! 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.110 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:35:54.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:35:54.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T06:35:54.111 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: pgmap v369: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.112 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: Upgrade: Finalizing container_image settings 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-27T06:35:54.113 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-27T06:35:54.114 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: Upgrade: Complete! 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.115 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:35:54.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:35:54.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:35:54.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:54.116 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:56.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:55 smithi131 ceph-mon[66259]: pgmap v370: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:56.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:55 smithi150 ceph-mon[52803]: pgmap v370: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:57.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:56 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:35:57.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:56 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:57.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:56 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:35:57.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:56 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:35:58.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:57 smithi131 ceph-mon[66259]: pgmap v371: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:35:58.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:57 smithi150 ceph-mon[52803]: pgmap v371: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:00.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:35:59 smithi131 ceph-mon[66259]: pgmap v372: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:00.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:35:59 smithi150 ceph-mon[52803]: pgmap v372: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:02.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:01 smithi150 ceph-mon[52803]: pgmap v373: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:02.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:01 smithi131 ceph-mon[66259]: pgmap v373: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:04.110 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:03 smithi150 ceph-mon[52803]: pgmap v374: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:36:04.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:03 smithi131 ceph-mon[66259]: pgmap v374: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:36:06.330 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:05 smithi131 ceph-mon[66259]: pgmap v375: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:06.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:05 smithi150 ceph-mon[52803]: pgmap v375: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:08.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:07 smithi131 ceph-mon[66259]: pgmap v376: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:08.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:07 smithi150 ceph-mon[52803]: pgmap v376: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:10.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:09 smithi131 ceph-mon[66259]: pgmap v377: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:10.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:09 smithi150 ceph-mon[52803]: pgmap v377: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:12.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:11 smithi131 ceph-mon[66259]: pgmap v378: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:12.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:11 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:36:12.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:11 smithi150 ceph-mon[52803]: pgmap v378: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:12.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:11 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:36:14.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:13 smithi131 ceph-mon[66259]: pgmap v379: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:36:14.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:13 smithi150 ceph-mon[52803]: pgmap v379: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:36:16.331 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:15 smithi131 ceph-mon[66259]: pgmap v380: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:16.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:15 smithi150 ceph-mon[52803]: pgmap v380: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:18.355 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:17 smithi131 ceph-mon[66259]: pgmap v381: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:18.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:17 smithi150 ceph-mon[52803]: pgmap v381: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:20.162 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:19 smithi150 ceph-mon[52803]: pgmap v382: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:20.223 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:19 smithi131 ceph-mon[66259]: pgmap v382: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:20.853 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph orch ps' 2024-02-27T06:36:21.105 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:20 smithi131 ceph-mon[66259]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:36:21.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:20 smithi150 ceph-mon[52803]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:36:22.211 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:21 smithi131 ceph-mon[66259]: pgmap v383: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:22.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:21 smithi150 ceph-mon[52803]: pgmap v383: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:22.943 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:36:22.943 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (57s) 35s ago 21m 14.9M - 0.25.0 c8568f914cd2 d21da279d6aa 2024-02-27T06:36:22.943 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (8m) 35s ago 21m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:36:22.943 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (8m) 88s ago 20m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:36:22.943 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (40s) 35s ago 21m 51.6M - 9.4.12 a0ea5e3c46d9 3286cc15d984 2024-02-27T06:36:22.943 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (2m) 35s ago 15m 36.4M - 19.0.0-1736-g635ee685 c4eb0759a73a 5b3c3fc0c3d9 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (2m) 88s ago 15m 13.9M - 19.0.0-1736-g635ee685 c4eb0759a73a 2e0e5aff32ee 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (11m) 35s ago 22m 516M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (10m) 88s ago 20m 453M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (10m) 35s ago 22m 63.0M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (9m) 88s ago 20m 48.6M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (2m) 35s ago 15m 69.1M - 5.7 c4eb0759a73a 5340eb3423e8 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (118s) 88s ago 15m 68.5M - 5.7 c4eb0759a73a 75f5b5dbf7dc 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (98s) 35s ago 21m 8799k - 1.5.0 0da6a335fe13 bf1053818375 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (91s) 88s ago 20m 3581k - 1.5.0 0da6a335fe13 e1c6d1fee8ba 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (7m) 35s ago 19m 420M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (7m) 35s ago 19m 377M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (6m) 35s ago 19m 374M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (5m) 35s ago 18m 430M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 3d240baa2aed 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (5m) 88s ago 18m 355M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 7dc53922c309 2024-02-27T06:36:22.944 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (4m) 88s ago 17m 324M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 6865822553f8 2024-02-27T06:36:22.945 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (3m) 88s ago 17m 307M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 1dedb9e4ac79 2024-02-27T06:36:22.945 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (3m) 88s ago 17m 317M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 50595e9edf32 2024-02-27T06:36:22.945 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (74s) 35s ago 20m 51.5M - 2.43.0 a07b618ecd1d c7dcd7362c89 2024-02-27T06:36:23.643 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph versions' 2024-02-27T06:36:24.199 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:23 smithi131 ceph-mon[66259]: pgmap v384: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:36:24.200 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:23 smithi131 ceph-mon[66259]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:36:24.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:23 smithi150 ceph-mon[52803]: pgmap v384: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:36:24.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:23 smithi150 ceph-mon[52803]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:36:25.819 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:36:25.819 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 14 2024-02-27T06:36:25.820 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:36:25.821 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:36:26.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:25 smithi131 ceph-mon[66259]: pgmap v385: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:26.104 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:25 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/200073079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:36:26.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:25 smithi150 ceph-mon[52803]: pgmap v385: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:26.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:25 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/200073079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:36:26.452 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-02-27T06:36:27.082 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:36:27.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:36:28.014 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:27 smithi131 ceph-mon[66259]: pgmap v386: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:28.265 INFO:teuthology.orchestra.run.smithi131.stdout:wait for servicemap items w/ changing names to refresh 2024-02-27T06:36:28.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:27 smithi150 ceph-mon[52803]: pgmap v386: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:28.923 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'sleep 60' 2024-02-27T06:36:30.194 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:29 smithi131 ceph-mon[66259]: pgmap v387: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:30.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:29 smithi150 ceph-mon[52803]: pgmap v387: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:32.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:31 smithi131 ceph-mon[66259]: pgmap v388: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:32.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:31 smithi150 ceph-mon[52803]: pgmap v388: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:34.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:33 smithi131 ceph-mon[66259]: pgmap v389: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:36:34.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:33 smithi150 ceph-mon[52803]: pgmap v389: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:36:36.331 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:35 smithi131 ceph-mon[66259]: pgmap v390: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:36.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:35 smithi150 ceph-mon[52803]: pgmap v390: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:38.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:37 smithi131 ceph-mon[66259]: pgmap v391: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:38.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:37 smithi150 ceph-mon[52803]: pgmap v391: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:40.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:39 smithi131 ceph-mon[66259]: pgmap v392: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:40.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:39 smithi150 ceph-mon[52803]: pgmap v392: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:42.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:41 smithi131 ceph-mon[66259]: pgmap v393: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:42.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:36:42.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:41 smithi150 ceph-mon[52803]: pgmap v393: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:42.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:36:44.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:43 smithi131 ceph-mon[66259]: pgmap v394: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:36:44.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:43 smithi150 ceph-mon[52803]: pgmap v394: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:36:46.330 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:45 smithi131 ceph-mon[66259]: pgmap v395: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:46.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:45 smithi150 ceph-mon[52803]: pgmap v395: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:48.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:47 smithi131 ceph-mon[66259]: pgmap v396: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:48.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:47 smithi150 ceph-mon[52803]: pgmap v396: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:50.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:49 smithi131 ceph-mon[66259]: pgmap v397: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:50.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:49 smithi150 ceph-mon[52803]: pgmap v397: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:52.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:51 smithi131 ceph-mon[66259]: pgmap v398: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:52.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:51 smithi150 ceph-mon[52803]: pgmap v398: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:54.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:53 smithi131 ceph-mon[66259]: pgmap v399: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:36:54.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:36:54.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:36:54.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:53 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:36:54.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:53 smithi150 ceph-mon[52803]: pgmap v399: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:36:54.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:36:54.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:36:54.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:53 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:36:55.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:36:55.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:36:55.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:36:55.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:36:55.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:36:55.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:36:55.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:36:55.360 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:36:56.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:56 smithi131 ceph-mon[66259]: pgmap v400: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:56.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:56 smithi150 ceph-mon[52803]: pgmap v400: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:57.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:36:57.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:36:58.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:36:58 smithi131 ceph-mon[66259]: pgmap v401: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:36:58.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:36:58 smithi150 ceph-mon[52803]: pgmap v401: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:00.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:00 smithi131 ceph-mon[66259]: pgmap v402: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:00.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:00 smithi150 ceph-mon[52803]: pgmap v402: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:02.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:02 smithi131 ceph-mon[66259]: pgmap v403: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:02.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:02 smithi150 ceph-mon[52803]: pgmap v403: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:04.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:04 smithi131 ceph-mon[66259]: pgmap v404: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:37:04.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:04 smithi150 ceph-mon[52803]: pgmap v404: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:37:06.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:06 smithi131 ceph-mon[66259]: pgmap v405: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:06.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:06 smithi150 ceph-mon[52803]: pgmap v405: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:08.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:08 smithi131 ceph-mon[66259]: pgmap v406: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:08.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:08 smithi150 ceph-mon[52803]: pgmap v406: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:10.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:10 smithi131 ceph-mon[66259]: pgmap v407: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:10.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:10 smithi150 ceph-mon[52803]: pgmap v407: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:12.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:12 smithi131 ceph-mon[66259]: pgmap v408: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:12.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:12 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:37:12.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:12 smithi150 ceph-mon[52803]: pgmap v408: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:12.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:12 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:37:14.603 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:14 smithi131 ceph-mon[66259]: pgmap v409: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:37:14.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:14 smithi150 ceph-mon[52803]: pgmap v409: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:37:16.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:16 smithi131 ceph-mon[66259]: pgmap v410: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:16.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:16 smithi150 ceph-mon[52803]: pgmap v410: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:18.604 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:18 smithi131 ceph-mon[66259]: pgmap v411: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:18.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:18 smithi150 ceph-mon[52803]: pgmap v411: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:19.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:19 smithi131 ceph-mon[66259]: pgmap v412: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:19.859 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:19 smithi150 ceph-mon[52803]: pgmap v412: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:22.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:21 smithi131 ceph-mon[66259]: pgmap v413: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:22.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:21 smithi150 ceph-mon[52803]: pgmap v413: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:24.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:23 smithi131 ceph-mon[66259]: pgmap v414: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:37:24.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:23 smithi150 ceph-mon[52803]: pgmap v414: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:37:26.329 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:25 smithi131 ceph-mon[66259]: pgmap v415: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:26.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:25 smithi150 ceph-mon[52803]: pgmap v415: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:27.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:26 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:37:27.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:26 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:37:28.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:27 smithi131 ceph-mon[66259]: pgmap v416: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:28.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:27 smithi150 ceph-mon[52803]: pgmap v416: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:30.354 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:29 smithi131 ceph-mon[66259]: pgmap v417: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:30.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:29 smithi150 ceph-mon[52803]: pgmap v417: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:31.310 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph orch ps' 2024-02-27T06:37:32.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:31 smithi131 ceph-mon[66259]: pgmap v418: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:32.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:31 smithi150 ceph-mon[52803]: pgmap v418: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:33.417 INFO:teuthology.orchestra.run.smithi131.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-27T06:37:33.417 INFO:teuthology.orchestra.run.smithi131.stdout:alertmanager.smithi131 smithi131 *:9093,9094 running (2m) 105s ago 22m 14.9M - 0.25.0 c8568f914cd2 d21da279d6aa 2024-02-27T06:37:33.417 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi131 smithi131 running (9m) 105s ago 22m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 4e88133cc2b3 2024-02-27T06:37:33.417 INFO:teuthology.orchestra.run.smithi131.stdout:crash.smithi150 smithi150 running (9m) 2m ago 21m 6660k - 19.0.0-1736-g635ee685 c4eb0759a73a 265986168bc9 2024-02-27T06:37:33.417 INFO:teuthology.orchestra.run.smithi131.stdout:grafana.smithi131 smithi131 *:3000 running (111s) 105s ago 22m 51.6M - 9.4.12 a0ea5e3c46d9 3286cc15d984 2024-02-27T06:37:33.417 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi131.mhuwwb smithi131 running (3m) 105s ago 16m 36.4M - 19.0.0-1736-g635ee685 c4eb0759a73a 5b3c3fc0c3d9 2024-02-27T06:37:33.417 INFO:teuthology.orchestra.run.smithi131.stdout:mds.foofs.smithi150.vsnjhu smithi150 running (3m) 2m ago 16m 13.9M - 19.0.0-1736-g635ee685 c4eb0759a73a 2e0e5aff32ee 2024-02-27T06:37:33.417 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi131.frtgqo smithi131 *:8443,9283,8765 running (12m) 105s ago 23m 516M - 19.0.0-1736-g635ee685 c4eb0759a73a b5fc542aebcf 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:mgr.smithi150.vvgbyr smithi150 *:8443,9283,8765 running (11m) 2m ago 21m 453M - 19.0.0-1736-g635ee685 c4eb0759a73a 885662153e55 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi131 smithi131 running (11m) 105s ago 23m 63.0M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a e5901645d10a 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:mon.smithi150 smithi150 running (10m) 2m ago 21m 48.6M 2048M 19.0.0-1736-g635ee685 c4eb0759a73a 61da9fd92290 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.0.0.smithi131.cezbzx smithi131 *:2049 running (3m) 105s ago 16m 69.1M - 5.7 c4eb0759a73a 5340eb3423e8 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:nfs.foo.1.0.smithi150.xgkkez smithi150 *:2049 running (3m) 2m ago 16m 68.5M - 5.7 c4eb0759a73a 75f5b5dbf7dc 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi131 smithi131 *:9100 running (2m) 105s ago 22m 8799k - 1.5.0 0da6a335fe13 bf1053818375 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:node-exporter.smithi150 smithi150 *:9100 running (2m) 2m ago 21m 3581k - 1.5.0 0da6a335fe13 e1c6d1fee8ba 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:osd.0 smithi131 running (9m) 105s ago 21m 420M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 73ab15ca6f02 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:osd.1 smithi131 running (8m) 105s ago 20m 377M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a be4665f197cb 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:osd.2 smithi131 running (7m) 105s ago 20m 374M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 63a1c6f33f95 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:osd.3 smithi131 running (6m) 105s ago 19m 430M 1936M 19.0.0-1736-g635ee685 c4eb0759a73a 3d240baa2aed 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:osd.4 smithi150 running (6m) 2m ago 19m 355M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 7dc53922c309 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:osd.5 smithi150 running (5m) 2m ago 18m 324M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 6865822553f8 2024-02-27T06:37:33.418 INFO:teuthology.orchestra.run.smithi131.stdout:osd.6 smithi150 running (4m) 2m ago 18m 307M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 1dedb9e4ac79 2024-02-27T06:37:33.419 INFO:teuthology.orchestra.run.smithi131.stdout:osd.7 smithi150 running (4m) 2m ago 18m 317M 2704M 19.0.0-1736-g635ee685 c4eb0759a73a 50595e9edf32 2024-02-27T06:37:33.419 INFO:teuthology.orchestra.run.smithi131.stdout:prometheus.smithi131 smithi131 *:9095 running (2m) 105s ago 21m 51.5M - 2.43.0 a07b618ecd1d c7dcd7362c89 2024-02-27T06:37:34.082 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:33 smithi131 ceph-mon[66259]: pgmap v419: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:37:34.106 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph orch upgrade status' 2024-02-27T06:37:34.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:33 smithi150 ceph-mon[52803]: pgmap v419: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:37:35.103 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:34 smithi131 ceph-mon[66259]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:37:35.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:34 smithi150 ceph-mon[52803]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:37:36.103 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:35 smithi131 ceph-mon[66259]: pgmap v420: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:36.109 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:37:36.109 INFO:teuthology.orchestra.run.smithi131.stdout: "target_image": null, 2024-02-27T06:37:36.109 INFO:teuthology.orchestra.run.smithi131.stdout: "in_progress": false, 2024-02-27T06:37:36.109 INFO:teuthology.orchestra.run.smithi131.stdout: "which": "", 2024-02-27T06:37:36.109 INFO:teuthology.orchestra.run.smithi131.stdout: "services_complete": [], 2024-02-27T06:37:36.109 INFO:teuthology.orchestra.run.smithi131.stdout: "progress": null, 2024-02-27T06:37:36.110 INFO:teuthology.orchestra.run.smithi131.stdout: "message": "", 2024-02-27T06:37:36.110 INFO:teuthology.orchestra.run.smithi131.stdout: "is_paused": false 2024-02-27T06:37:36.110 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:37:36.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:35 smithi150 ceph-mon[52803]: pgmap v420: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:36.746 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph health detail' 2024-02-27T06:37:37.041 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:36 smithi131 ceph-mon[66259]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:37:37.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:36 smithi150 ceph-mon[52803]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:37:37.965 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:37 smithi131 ceph-mon[66259]: pgmap v421: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:38.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:37 smithi150 ceph-mon[52803]: pgmap v421: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:38.755 INFO:teuthology.orchestra.run.smithi131.stdout:HEALTH_OK 2024-02-27T06:37:39.064 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:38 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/2810986008' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:37:39.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:38 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/2810986008' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-27T06:37:39.379 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph versions' 2024-02-27T06:37:40.018 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:39 smithi131 ceph-mon[66259]: pgmap v422: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:40.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:39 smithi150 ceph-mon[52803]: pgmap v422: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:41.476 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-02-27T06:37:41.476 INFO:teuthology.orchestra.run.smithi131.stdout: "mon": { 2024-02-27T06:37:41.476 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: "mgr": { 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: "osd": { 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 8 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: "mds": { 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 2 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: "overall": { 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)": 14 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:37:41.477 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-02-27T06:37:42.103 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:41 smithi131 ceph-mon[66259]: pgmap v423: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:42.103 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:41 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/1299528636' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:37:42.103 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:41 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:37:42.155 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-02-27T06:37:42.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:41 smithi150 ceph-mon[52803]: pgmap v423: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:42.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:41 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/1299528636' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:37:42.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:41 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:37:44.103 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:43 smithi131 ceph-mon[66259]: pgmap v424: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:37:44.242 INFO:teuthology.orchestra.run.smithi131.stdout:true 2024-02-27T06:37:44.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:43 smithi150 ceph-mon[52803]: pgmap v424: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:37:44.939 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -e sha1=635ee6850c57c98cc6ba932cade45f9cc015aa32 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-02-27T06:37:44.964 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:44 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/961346398' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:37:45.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:44 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/961346398' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:37:46.230 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:45 smithi131 ceph-mon[66259]: pgmap v425: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:46.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:45 smithi150 ceph-mon[52803]: pgmap v425: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:47.221 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph version 19.0.0-1736-g635ee685 (635ee6850c57c98cc6ba932cade45f9cc015aa32) squid (dev)" 2024-02-27T06:37:47.914 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-02-27T06:37:47.914 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-02-27T06:37:47.914 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- ceph orch ls -f json 2024-02-27T06:37:48.248 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:47 smithi131 ceph-mon[66259]: pgmap v426: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:48.249 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:47 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/1645585028' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:37:48.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:47 smithi150 ceph-mon[52803]: pgmap v426: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:48.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:47 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/1645585028' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-27T06:37:49.922 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-02-27T06:37:49.922 INFO:teuthology.orchestra.run.smithi131.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-02-27T06:14:23.661249Z", "last_refresh": "2024-02-27T06:35:47.892779Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-02-27T06:14:17.375793Z", "last_refresh": "2024-02-27T06:34:54.522038Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-02-27T06:14:20.285100Z", "last_refresh": "2024-02-27T06:35:47.893093Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-02-27T06:20:44.960815Z", "last_refresh": "2024-02-27T06:34:54.522647Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-02-27T06:14:15.518198Z", "last_refresh": "2024-02-27T06:34:54.522140Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi131:172.21.15.131=smithi131", "smithi150:172.21.15.150=smithi150"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-02-27T06:15:04.040710Z", "last_refresh": "2024-02-27T06:34:54.522217Z", "running": 2, "size": 2}}, {"events": ["2024-02-27T06:36:54.280579Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-02-27T06:21:05.932533Z", "last_refresh": "2024-02-27T06:34:54.522716Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-02-27T06:14:21.429969Z", "last_refresh": "2024-02-27T06:34:54.522306Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e", "last_refresh": "2024-02-27T06:34:54.522375Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-02-27T06:14:18.623960Z", "last_refresh": "2024-02-27T06:35:47.893408Z", "ports": [9095], "running": 1, "size": 1}}] 2024-02-27T06:37:50.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:49 smithi131 ceph-mon[66259]: pgmap v427: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:50.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:49 smithi150 ceph-mon[52803]: pgmap v427: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:50.570 INFO:tasks.cephadm:nfs.foo has 2/2 2024-02-27T06:37:50.571 DEBUG:teuthology.parallel:result is None 2024-02-27T06:37:50.571 INFO:teuthology.run_tasks:Running task vip.exec... 2024-02-27T06:37:50.581 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:37:50.581 DEBUG:teuthology.orchestra.run.smithi131:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-02-27T06:37:50.610 INFO:teuthology.orchestra.run.smithi131.stderr:+ umount /mnt/foo 2024-02-27T06:37:50.621 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-27T06:37:50.631 WARNING:tasks.cephadm:rctx={'ctx': {'help': False, 'version': False, 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'block': False, 'lock': False, 'suite_path': None, 'archive': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'machine_type': 'smithi', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'owner': 'scheduled_adking@teuthology', 'os_version': None, 'verbose': True, 'os_type': None, 'interactive_on_error': False, '': True, 'config': {'archive_path': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'branch': 'wip-adk-testing-2024-02-26-1600', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'email': 'adking@redhat.com', 'first_in_suite': False, 'job_id': '7574463', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-adk-testing-2024-02-26-1600'}, 'ceph': {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-adk-testing-2024-02-26-1600', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'owner': 'scheduled_adking@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 3790, 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'sleep_before_teardown': 0, 'subset': '1/14', 'suite': 'orch:cephadm', 'suite_branch': 'wip-adk-testing-2024-02-26-1600', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'targets': {'smithi131.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyMMU2WmPTrG0nEyQ2NmvUMXtb5sj/oZhk6zkIj6rVET3wh7KaevTVGclb2I20iC0f0mf/os1Vdr5fuzKhXUrU=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1EuQQJmp6qcOuc+G8GQiW/tx9dd8YBP2iomrl9Czw/K/0oLGOOuefBVJ8PZ2KxfCk/0AouJ1k7OKERdbVOv+k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'cephadm': {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'cluster': 'ceph', 'cephadm_mode': 'root'}}, {'cephadm.shell': {'host.a': ['ceph orch status', 'ceph orch ps', 'ceph orch ls', 'ceph orch host ls', 'ceph orch device ls']}}, {'vip.exec': {'all-hosts': ['systemctl stop nfs-server']}}, {'cephadm.shell': {'host.a': ['ceph fs volume create foofs']}}, {'cephadm.wait_for_service': {'service': 'mds.foofs'}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2', 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake', 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done']}}, {'vip.exec': {'host.a': ['mkdir /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done', 'echo test > /mnt/foo/testfile', 'sync']}}, {'parallel': ['upgrade-tasks', 'workload-tasks']}, {'vip.exec': {'host.a': ['umount /mnt/foo']}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster ls | grep foo', 'ceph nfs export ls foo --detailed', 'rados -p .nfs --all ls -']}}, {'cephadm.shell': {'host.a': ['set -ex\n[[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]]\n']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-27_05:08:20', 'tube': 'smithi', 'upgrade-tasks': {'sequential': [{'cephadm.shell': {'host.a': ['ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph mgr module enable nfs --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}}, {'cephadm.shell': {'host.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph orch upgrade status', 'ceph health detail', 'ceph versions', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1"]}}, {'cephadm.wait_for_service': {'service': 'nfs.foo'}}]}, 'user': 'adking', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885', 'workload-tasks': {'sequential': [{'exec': {'host.a': ['cd /mnt/foo && dbench 5 -t 600 || true', 'umount /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done', 'cd /mnt/foo && dbench 5 -t 5']}}]}}, 'summary': {'success': True, 'owner': 'scheduled_adking@teuthology', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}'}, 'cluster': Cluster(remotes=[[Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.smithi131']], [Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.smithi150']]]), 'ceph': {'ceph': Namespace(admin_keyring=b'[client.admin]\n\tkey = AQCQfd1ldPHTGRAAO9qJUosu3KZYG3ESuy/Q6w==\n', bootstrap_remote=Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), bootstrapped=False, conf=ConfigObj({'global': {'log_to_file': 'true', 'log_to_stderr': 'false', 'log to journald': 'false', 'mon cluster log to file': 'true', 'mon cluster log file level': 'debug', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'debug asserts on shutdown': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'osd_pool_default_pg_autoscale_mode': 'off', 'mon allow pool delete': 'true', 'fsid': '3fa4c766-d537-11ee-95c3-87774f69a715'}, 'osd': {'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'osd memory target autotune': 'true', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug pg log writeout': 'true', 'osd debug verify cached snaps': 'true', 'osd debug verify missing on start': 'true', 'osd debug misdirected ops': 'true', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True, 'bdev debug aio': 'true', 'osd sloppy crc': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug mgr': 20, 'debug ms': 1}, 'mon': {'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'client.rgw': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true'}}), config_file='# minimal ceph.conf for 3fa4c766-d537-11ee-95c3-87774f69a715\n[global]\n\tfsid = 3fa4c766-d537-11ee-95c3-87774f69a715\n\tmon_host = [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]\n', first_mon='smithi131', first_mon_role='mon.smithi131', fsid='3fa4c766-d537-11ee-95c3-87774f69a715', image='quay.io/ceph/ceph:v17.2.0', mon_keyring=b'[mon.]\n\tkey = AQCPfd1l3PmOHxAA+kWw7vIRwPoCEi4ziUQ3cA==\n\tcaps mon = "allow *"\n', mons={'mon.smithi131': '172.21.15.131', 'mon.smithi150': '172.21.15.150'}, roleless=True, thrashers=[])}, 'cephadm': '/home/ubuntu/cephtest/cephadm', 'daemons': , 'managers': {'ceph': }, '_jinja_env': }, 'config': {'host.a': ['ceph nfs cluster ls | grep foo', 'ceph nfs export ls foo --detailed', 'rados -p .nfs --all ls -']}, 'cluster_name': ''} 2024-02-27T06:37:50.631 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:37:50.632 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-02-27T06:37:50.965 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:50 smithi131 ceph-mon[66259]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:37:51.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:50 smithi150 ceph-mon[52803]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-27T06:37:52.250 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:51 smithi131 ceph-mon[66259]: pgmap v428: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:52.260 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:51 smithi150 ceph-mon[52803]: pgmap v428: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:52.603 INFO:teuthology.orchestra.run.smithi131.stdout: "foo" 2024-02-27T06:37:53.188 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'ceph nfs export ls foo --detailed' 2024-02-27T06:37:54.103 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:53 smithi131 ceph-mon[66259]: from='client.34666 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:37:54.103 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:53 smithi131 ceph-mon[66259]: pgmap v429: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:37:54.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:53 smithi150 ceph-mon[52803]: from='client.34666 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:37:54.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:53 smithi150 ceph-mon[52803]: pgmap v429: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-27T06:37:55.103 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:54 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:37:55.278 INFO:teuthology.orchestra.run.smithi131.stdout:[ 2024-02-27T06:37:55.278 INFO:teuthology.orchestra.run.smithi131.stdout: { 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: "access_type": "RW", 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: "clients": [], 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: "cluster_id": "foo", 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: "export_id": 1, 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: "fsal": { 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: "fs_name": "foofs", 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "CEPH", 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: "user_id": "nfs.foo.1" 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: "path": "/", 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: "protocols": [ 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: 4 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2024-02-27T06:37:55.279 INFO:teuthology.orchestra.run.smithi131.stdout: "pseudo": "/fake", 2024-02-27T06:37:55.280 INFO:teuthology.orchestra.run.smithi131.stdout: "security_label": true, 2024-02-27T06:37:55.280 INFO:teuthology.orchestra.run.smithi131.stdout: "squash": "none", 2024-02-27T06:37:55.280 INFO:teuthology.orchestra.run.smithi131.stdout: "transports": [ 2024-02-27T06:37:55.280 INFO:teuthology.orchestra.run.smithi131.stdout: "TCP" 2024-02-27T06:37:55.280 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-02-27T06:37:55.280 INFO:teuthology.orchestra.run.smithi131.stdout: } 2024-02-27T06:37:55.280 INFO:teuthology.orchestra.run.smithi131.stdout:] 2024-02-27T06:37:55.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:54 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-27T06:37:55.903 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'rados -p .nfs --all ls -' 2024-02-27T06:37:56.209 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:55 smithi131 ceph-mon[66259]: pgmap v430: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:56.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:55 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:37:56.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:55 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:37:56.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:55 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:37:56.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:55 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:37:56.210 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:55 smithi131 ceph-mon[66259]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:37:56.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:55 smithi150 ceph-mon[52803]: pgmap v430: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:56.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:55 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-27T06:37:56.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:55 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-27T06:37:56.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:55 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:37:56.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:55 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' 2024-02-27T06:37:56.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:55 smithi150 ceph-mon[52803]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-27T06:37:57.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:57 smithi131 ceph-mon[66259]: mgrmap e43: smithi131.frtgqo(active, since 11m), standbys: smithi150.vvgbyr 2024-02-27T06:37:57.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:57 smithi131 ceph-mon[66259]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:37:57.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:57 smithi150 ceph-mon[52803]: mgrmap e43: smithi131.frtgqo(active, since 11m), standbys: smithi150.vvgbyr 2024-02-27T06:37:57.609 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:57 smithi150 ceph-mon[52803]: from='mgr.34102 172.21.15.131:0/2119659400' entity='mgr.smithi131.frtgqo' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-27T06:37:57.917 INFO:teuthology.orchestra.run.smithi131.stdout:foo rec-0000000000000004:nfs.foo.1 2024-02-27T06:37:57.917 INFO:teuthology.orchestra.run.smithi131.stdout:foo grace 2024-02-27T06:37:57.917 INFO:teuthology.orchestra.run.smithi131.stdout:foo rec-0000000000000004:nfs.foo.0 2024-02-27T06:37:57.917 INFO:teuthology.orchestra.run.smithi131.stdout:foo conf-nfs.foo 2024-02-27T06:37:57.917 INFO:teuthology.orchestra.run.smithi131.stdout:foo export-1 2024-02-27T06:37:58.454 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:37:58 smithi131 ceph-mon[66259]: pgmap v431: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:37:58.483 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-27T06:37:58.493 WARNING:tasks.cephadm:rctx={'ctx': {'help': False, 'version': False, 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'block': False, 'lock': False, 'suite_path': None, 'archive': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'machine_type': 'smithi', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'owner': 'scheduled_adking@teuthology', 'os_version': None, 'verbose': True, 'os_type': None, 'interactive_on_error': False, '': True, 'config': {'archive_path': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'branch': 'wip-adk-testing-2024-02-26-1600', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}', 'email': 'adking@redhat.com', 'first_in_suite': False, 'job_id': '7574463', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-adk-testing-2024-02-26-1600'}, 'ceph': {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-adk-testing-2024-02-26-1600', 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32'}}, 'owner': 'scheduled_adking@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 3790, 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'sleep_before_teardown': 0, 'subset': '1/14', 'suite': 'orch:cephadm', 'suite_branch': 'wip-adk-testing-2024-02-26-1600', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_635ee6850c57c98cc6ba932cade45f9cc015aa32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'targets': {'smithi131.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyMMU2WmPTrG0nEyQ2NmvUMXtb5sj/oZhk6zkIj6rVET3wh7KaevTVGclb2I20iC0f0mf/os1Vdr5fuzKhXUrU=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1EuQQJmp6qcOuc+G8GQiW/tx9dd8YBP2iomrl9Czw/K/0oLGOOuefBVJ8PZ2KxfCk/0AouJ1k7OKERdbVOv+k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'cephadm': {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '635ee6850c57c98cc6ba932cade45f9cc015aa32', 'cluster': 'ceph', 'cephadm_mode': 'root'}}, {'cephadm.shell': {'host.a': ['ceph orch status', 'ceph orch ps', 'ceph orch ls', 'ceph orch host ls', 'ceph orch device ls']}}, {'vip.exec': {'all-hosts': ['systemctl stop nfs-server']}}, {'cephadm.shell': {'host.a': ['ceph fs volume create foofs']}}, {'cephadm.wait_for_service': {'service': 'mds.foofs'}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2', 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake', 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done']}}, {'vip.exec': {'host.a': ['mkdir /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done', 'echo test > /mnt/foo/testfile', 'sync']}}, {'parallel': ['upgrade-tasks', 'workload-tasks']}, {'vip.exec': {'host.a': ['umount /mnt/foo']}}, {'cephadm.shell': {'host.a': ['ceph nfs cluster ls | grep foo', 'ceph nfs export ls foo --detailed', 'rados -p .nfs --all ls -']}}, {'cephadm.shell': {'host.a': ['set -ex\n[[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]]\n']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-27_05:08:20', 'tube': 'smithi', 'upgrade-tasks': {'sequential': [{'cephadm.shell': {'host.a': ['ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force', 'ceph config set global log_to_journald false --force', 'ceph mgr module enable nfs --force', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1']}}, {'cephadm.shell': {'host.a': ["while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done", 'ceph orch ps', 'ceph versions', 'echo "wait for servicemap items w/ changing names to refresh"', 'sleep 60', 'ceph orch ps', 'ceph orch upgrade status', 'ceph health detail', 'ceph versions', "ceph versions | jq -e '.overall | length == 1'", "ceph versions | jq -e '.overall | keys' | grep $sha1"]}}, {'cephadm.wait_for_service': {'service': 'nfs.foo'}}]}, 'user': 'adking', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885', 'workload-tasks': {'sequential': [{'exec': {'host.a': ['cd /mnt/foo && dbench 5 -t 600 || true', 'umount /mnt/foo', 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done', 'cd /mnt/foo && dbench 5 -t 5']}}]}}, 'summary': {'success': True, 'owner': 'scheduled_adking@teuthology', 'description': 'orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final}'}, 'cluster': Cluster(remotes=[[Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.smithi131']], [Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.smithi150']]]), 'ceph': {'ceph': Namespace(admin_keyring=b'[client.admin]\n\tkey = AQCQfd1ldPHTGRAAO9qJUosu3KZYG3ESuy/Q6w==\n', bootstrap_remote=Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), bootstrapped=False, conf=ConfigObj({'global': {'log_to_file': 'true', 'log_to_stderr': 'false', 'log to journald': 'false', 'mon cluster log to file': 'true', 'mon cluster log file level': 'debug', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'debug asserts on shutdown': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'osd_pool_default_pg_autoscale_mode': 'off', 'mon allow pool delete': 'true', 'fsid': '3fa4c766-d537-11ee-95c3-87774f69a715'}, 'osd': {'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'osd memory target autotune': 'true', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug pg log writeout': 'true', 'osd debug verify cached snaps': 'true', 'osd debug verify missing on start': 'true', 'osd debug misdirected ops': 'true', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True, 'bdev debug aio': 'true', 'osd sloppy crc': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug mgr': 20, 'debug ms': 1}, 'mon': {'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'client.rgw': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true'}}), config_file='# minimal ceph.conf for 3fa4c766-d537-11ee-95c3-87774f69a715\n[global]\n\tfsid = 3fa4c766-d537-11ee-95c3-87774f69a715\n\tmon_host = [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0]\n', first_mon='smithi131', first_mon_role='mon.smithi131', fsid='3fa4c766-d537-11ee-95c3-87774f69a715', image='quay.io/ceph/ceph:v17.2.0', mon_keyring=b'[mon.]\n\tkey = AQCPfd1l3PmOHxAA+kWw7vIRwPoCEi4ziUQ3cA==\n\tcaps mon = "allow *"\n', mons={'mon.smithi131': '172.21.15.131', 'mon.smithi150': '172.21.15.150'}, roleless=True, thrashers=[])}, 'cephadm': '/home/ubuntu/cephtest/cephadm', 'daemons': , 'managers': {'ceph': }, '_jinja_env': }, 'config': {'host.a': ['set -ex\n[[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]]\n']}, 'cluster_name': ''} 2024-02-27T06:37:58.493 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:37:58.494 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 -- bash -c 'set -ex 2024-02-27T06:37:58.494 DEBUG:teuthology.orchestra.run.smithi131:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-02-27T06:37:58.494 DEBUG:teuthology.orchestra.run.smithi131:> ' 2024-02-27T06:37:58.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:37:58 smithi150 ceph-mon[52803]: pgmap v431: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-27T06:38:00.291 INFO:teuthology.orchestra.run.smithi131.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-02-27T06:38:00.353 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:38:00 smithi131 ceph-mon[66259]: pgmap v432: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-27T06:38:00.514 INFO:teuthology.orchestra.run.smithi131.stderr:+ [[ 6 -gt 2 ]] 2024-02-27T06:38:00.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:38:00 smithi150 ceph-mon[52803]: pgmap v432: 97 pgs: 97 active+clean; 232 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-27T06:38:01.110 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-27T06:38:01.122 INFO:tasks.cephadm:Teardown begin 2024-02-27T06:38:01.122 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-27T06:38:01.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-27T06:38:01.187 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-27T06:38:01.187 DEBUG:teuthology.orchestra.run.smithi131:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-27T06:38:01.208 DEBUG:teuthology.orchestra.run.smithi150:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-27T06:38:01.245 INFO:tasks.cephadm:Stopping all daemons... 2024-02-27T06:38:01.245 INFO:tasks.cephadm.mon.smithi131:Stopping mon.smithi131... 2024-02-27T06:38:01.245 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi131 2024-02-27T06:38:01.591 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:38:01 smithi131 ceph-mon[66259]: from='client.? 172.21.15.131:0/1110931691' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-02-27T06:38:01.591 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:38:01 smithi131 systemd[1]: Stopping Ceph mon.smithi131 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:38:01.608 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:38:01 smithi150 ceph-mon[52803]: from='client.? 172.21.15.131:0/1110931691' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-02-27T06:38:01.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:38:01 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131[66255]: 2024-02-27T06:38:01.583+0000 7f174bf92700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi131 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:38:01.853 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:38:01 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131[66255]: 2024-02-27T06:38:01.583+0000 7f174bf92700 -1 mon.smithi131@0(leader) e3 *** Got Signal Terminated *** 2024-02-27T06:38:02.142 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:38:01 smithi131 podman[112526]: 2024-02-27 06:38:01.848773671 +0000 UTC m=+0.422379796 container died e5901645d10a3889439178583676be37341400fe297523827b332aa80c45efde (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-27T06:38:02.501 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:38:02 smithi131 podman[112526]: 2024-02-27 06:38:02.242892793 +0000 UTC m=+0.816498924 container cleanup e5901645d10a3889439178583676be37341400fe297523827b332aa80c45efde (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:38:02.501 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Feb 27 06:38:02 smithi131 bash[112526]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi131 2024-02-27T06:38:02.676 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi131.service' 2024-02-27T06:38:02.724 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T06:38:02.724 INFO:tasks.cephadm.mon.smithi131:Stopped mon.smithi131 2024-02-27T06:38:02.725 INFO:tasks.cephadm.mon.smithi150:Stopping mon.smithi150... 2024-02-27T06:38:02.725 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi150 2024-02-27T06:38:03.031 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:38:02 smithi150 systemd[1]: Stopping Ceph mon.smithi150 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:38:03.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:38:03 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150[52799]: 2024-02-27T06:38:03.023+0000 7ff758686700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi150 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:38:03.358 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:38:03 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150[52799]: 2024-02-27T06:38:03.023+0000 7ff758686700 -1 mon.smithi150@1(peon) e3 *** Got Signal Terminated *** 2024-02-27T06:38:03.359 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:38:03 smithi150 podman[78641]: 2024-02-27 06:38:03.269004073 +0000 UTC m=+0.348868601 container died 61da9fd9229099cc2527c8470cb148ff1ef6cf0019eb5794d5fa4d08ef2194a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) 2024-02-27T06:38:03.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:38:03 smithi150 podman[78641]: 2024-02-27 06:38:03.723705785 +0000 UTC m=+0.803570360 container cleanup 61da9fd9229099cc2527c8470cb148ff1ef6cf0019eb5794d5fa4d08ef2194a7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:38:03.858 INFO:journalctl@ceph.mon.smithi150.smithi150.stdout:Feb 27 06:38:03 smithi150 bash[78641]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-mon-smithi150 2024-02-27T06:38:04.200 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@mon.smithi150.service' 2024-02-27T06:38:04.247 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T06:38:04.247 INFO:tasks.cephadm.mon.smithi150:Stopped mon.smithi150 2024-02-27T06:38:04.247 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-27T06:38:04.247 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.0 2024-02-27T06:38:04.546 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:04 smithi131 systemd[1]: Stopping Ceph osd.0 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:38:04.853 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:04 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0[72420]: 2024-02-27T06:38:04.538+0000 7fae118d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:38:04.853 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:04 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0[72420]: 2024-02-27T06:38:04.538+0000 7fae118d0700 -1 osd.0 156 *** Got signal Terminated *** 2024-02-27T06:38:04.853 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:04 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0[72420]: 2024-02-27T06:38:04.538+0000 7fae118d0700 -1 osd.0 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:38:10.331 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:10 smithi131 podman[112690]: 2024-02-27 06:38:10.001266976 +0000 UTC m=+5.563652325 container died 73ab15ca6f02c3d349c707d9736d218fdd0739f0820627cce57800c3d10ac44b (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222) 2024-02-27T06:38:10.603 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:10 smithi131 podman[112690]: 2024-02-27 06:38:10.431328469 +0000 UTC m=+5.993713821 container cleanup 73ab15ca6f02c3d349c707d9736d218fdd0739f0820627cce57800c3d10ac44b (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-02-27T06:38:10.603 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:10 smithi131 bash[112690]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0 2024-02-27T06:38:11.103 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:10 smithi131 podman[112707]: 2024-02-27 06:38:10.65011195 +0000 UTC m=+0.651485084 container remove 73ab15ca6f02c3d349c707d9736d218fdd0739f0820627cce57800c3d10ac44b (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-27T06:38:11.419 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:11 smithi131 podman[112807]: 2024-02-27 06:38:11.037421746 +0000 UTC m=+0.019213982 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:38:11.419 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:11 smithi131 podman[112807]: 2024-02-27 06:38:11.134605295 +0000 UTC m=+0.116397527 container create d28bac1e29553443f4b1ee64580048dc397ee10673466a8cc9f7c2a18f5ed7ae (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True) 2024-02-27T06:38:11.419 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:11 smithi131 podman[112807]: 2024-02-27 06:38:11.412525741 +0000 UTC m=+0.394317970 container init d28bac1e29553443f4b1ee64580048dc397ee10673466a8cc9f7c2a18f5ed7ae (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True) 2024-02-27T06:38:11.722 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:11 smithi131 podman[112807]: 2024-02-27 06:38:11.416000588 +0000 UTC m=+0.397792825 container start d28bac1e29553443f4b1ee64580048dc397ee10673466a8cc9f7c2a18f5ed7ae (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS) 2024-02-27T06:38:11.722 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:11 smithi131 podman[112807]: 2024-02-27 06:38:11.521153511 +0000 UTC m=+0.502945749 container attach d28bac1e29553443f4b1ee64580048dc397ee10673466a8cc9f7c2a18f5ed7ae (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-27T06:38:12.074 INFO:journalctl@ceph.osd.0.smithi131.stdout:Feb 27 06:38:11 smithi131 podman[112807]: 2024-02-27 06:38:11.715418334 +0000 UTC m=+0.697210556 container died d28bac1e29553443f4b1ee64580048dc397ee10673466a8cc9f7c2a18f5ed7ae (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:38:12.390 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.0.service' 2024-02-27T06:38:12.434 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T06:38:12.434 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-27T06:38:12.434 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-27T06:38:12.434 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.1 2024-02-27T06:38:12.761 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:12 smithi131 systemd[1]: Stopping Ceph osd.1 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:38:13.103 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:12 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[75036]: 2024-02-27T06:38:12.753+0000 7f3753dc3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:38:13.103 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:12 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[75036]: 2024-02-27T06:38:12.753+0000 7f3753dc3700 -1 osd.1 156 *** Got signal Terminated *** 2024-02-27T06:38:13.103 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:12 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1[75036]: 2024-02-27T06:38:12.753+0000 7f3753dc3700 -1 osd.1 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:38:18.423 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:18 smithi131 podman[112914]: 2024-02-27 06:38:18.128725261 +0000 UTC m=+5.480457095 container died be4665f197cb4ae33a66fbbb1f04d744fc982c7f1e880d8360af92a1f0ec995f (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-27T06:38:18.778 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:18 smithi131 podman[112914]: 2024-02-27 06:38:18.495616265 +0000 UTC m=+5.847348099 container cleanup be4665f197cb4ae33a66fbbb1f04d744fc982c7f1e880d8360af92a1f0ec995f (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-02-27T06:38:18.778 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:18 smithi131 bash[112914]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1 2024-02-27T06:38:19.102 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:18 smithi131 podman[112931]: 2024-02-27 06:38:18.772030966 +0000 UTC m=+0.638161845 container remove be4665f197cb4ae33a66fbbb1f04d744fc982c7f1e880d8360af92a1f0ec995f (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:38:19.399 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:19 smithi131 podman[113025]: 2024-02-27 06:38:19.015691724 +0000 UTC m=+0.018308835 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:38:19.399 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:19 smithi131 podman[113025]: 2024-02-27 06:38:19.139908692 +0000 UTC m=+0.142525789 container create 8e86e2a4edebbd97e1f48c538f55801d75bb4a54b9a8187d03dcd34b7fae2132 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:38:19.399 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:19 smithi131 podman[113025]: 2024-02-27 06:38:19.392504416 +0000 UTC m=+0.395121515 container init 8e86e2a4edebbd97e1f48c538f55801d75bb4a54b9a8187d03dcd34b7fae2132 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=) 2024-02-27T06:38:19.697 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:19 smithi131 podman[113025]: 2024-02-27 06:38:19.395923314 +0000 UTC m=+0.398540418 container start 8e86e2a4edebbd97e1f48c538f55801d75bb4a54b9a8187d03dcd34b7fae2132 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:38:19.697 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:19 smithi131 podman[113025]: 2024-02-27 06:38:19.476016085 +0000 UTC m=+0.478633198 container attach 8e86e2a4edebbd97e1f48c538f55801d75bb4a54b9a8187d03dcd34b7fae2132 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:38:20.020 INFO:journalctl@ceph.osd.1.smithi131.stdout:Feb 27 06:38:19 smithi131 podman[113025]: 2024-02-27 06:38:19.690155507 +0000 UTC m=+0.692772608 container died 8e86e2a4edebbd97e1f48c538f55801d75bb4a54b9a8187d03dcd34b7fae2132 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:38:20.340 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.1.service' 2024-02-27T06:38:20.382 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T06:38:20.382 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-27T06:38:20.382 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-27T06:38:20.382 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.2 2024-02-27T06:38:20.602 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:20 smithi131 systemd[1]: Stopping Ceph osd.2 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:38:21.103 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:20 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2[77745]: 2024-02-27T06:38:20.717+0000 7f8af5f5b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:38:21.103 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:20 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2[77745]: 2024-02-27T06:38:20.717+0000 7f8af5f5b700 -1 osd.2 156 *** Got signal Terminated *** 2024-02-27T06:38:21.103 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:20 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2[77745]: 2024-02-27T06:38:20.717+0000 7f8af5f5b700 -1 osd.2 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:38:26.533 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:26 smithi131 podman[113132]: 2024-02-27 06:38:26.258980331 +0000 UTC m=+5.662211529 container died 63a1c6f33f95d2374a84449b58e453270b4811af862271ce462cd5d6a1a50876 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-27T06:38:26.852 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:26 smithi131 podman[113132]: 2024-02-27 06:38:26.608959923 +0000 UTC m=+6.012191132 container cleanup 63a1c6f33f95d2374a84449b58e453270b4811af862271ce462cd5d6a1a50876 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-27T06:38:26.853 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:26 smithi131 bash[113132]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2 2024-02-27T06:38:27.231 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:26 smithi131 podman[113183]: 2024-02-27 06:38:26.910745342 +0000 UTC m=+0.655622095 container remove 63a1c6f33f95d2374a84449b58e453270b4811af862271ce462cd5d6a1a50876 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-27T06:38:27.488 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:27 smithi131 podman[113272]: 2024-02-27 06:38:27.128104979 +0000 UTC m=+0.018182216 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:38:27.488 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:27 smithi131 podman[113272]: 2024-02-27 06:38:27.236787078 +0000 UTC m=+0.126864301 container create 0a8daa1548bea200cffdbdedae4f5c30a09d6456a8ab3fab91ad73f175f757d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True) 2024-02-27T06:38:27.488 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:27 smithi131 podman[113272]: 2024-02-27 06:38:27.481341518 +0000 UTC m=+0.371418750 container init 0a8daa1548bea200cffdbdedae4f5c30a09d6456a8ab3fab91ad73f175f757d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2) 2024-02-27T06:38:27.781 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:27 smithi131 podman[113272]: 2024-02-27 06:38:27.484833672 +0000 UTC m=+0.374910896 container start 0a8daa1548bea200cffdbdedae4f5c30a09d6456a8ab3fab91ad73f175f757d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux ) 2024-02-27T06:38:27.781 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:27 smithi131 podman[113272]: 2024-02-27 06:38:27.5644919 +0000 UTC m=+0.454569130 container attach 0a8daa1548bea200cffdbdedae4f5c30a09d6456a8ab3fab91ad73f175f757d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-27T06:38:28.103 INFO:journalctl@ceph.osd.2.smithi131.stdout:Feb 27 06:38:27 smithi131 podman[113272]: 2024-02-27 06:38:27.775061692 +0000 UTC m=+0.665138918 container died 0a8daa1548bea200cffdbdedae4f5c30a09d6456a8ab3fab91ad73f175f757d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-2-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0) 2024-02-27T06:38:28.544 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.2.service' 2024-02-27T06:38:28.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T06:38:28.586 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-27T06:38:28.586 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-27T06:38:28.586 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.3 2024-02-27T06:38:28.853 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:28 smithi131 systemd[1]: Stopping Ceph osd.3 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:38:29.352 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:28 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[80286]: 2024-02-27T06:38:28.938+0000 7fc38b059700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:38:29.353 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:28 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[80286]: 2024-02-27T06:38:28.938+0000 7fc38b059700 -1 osd.3 156 *** Got signal Terminated *** 2024-02-27T06:38:29.353 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:28 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[80286]: 2024-02-27T06:38:28.938+0000 7fc38b059700 -1 osd.3 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:38:30.734 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:30 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:30.458+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:31.853 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:31 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[80286]: 2024-02-27T06:38:31.469+0000 7fc383645700 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.225671+0000 front 2024-02-27T06:38:07.225808+0000 (oldest deadline 2024-02-27T06:38:31.325892+0000) 2024-02-27T06:38:31.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:31 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:31.483+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:31.858 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:31 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:31.495+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:05.951598+0000 front 2024-02-27T06:38:05.951697+0000 (oldest deadline 2024-02-27T06:38:31.251661+0000) 2024-02-27T06:38:32.852 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:32 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[80286]: 2024-02-27T06:38:32.456+0000 7fc383645700 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.225671+0000 front 2024-02-27T06:38:07.225808+0000 (oldest deadline 2024-02-27T06:38:31.325892+0000) 2024-02-27T06:38:32.858 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:32 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:32.486+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:05.951598+0000 front 2024-02-27T06:38:05.951697+0000 (oldest deadline 2024-02-27T06:38:31.251661+0000) 2024-02-27T06:38:32.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:32 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:32.454+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:33.827 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:33 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:33.451+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:05.951598+0000 front 2024-02-27T06:38:05.951697+0000 (oldest deadline 2024-02-27T06:38:31.251661+0000) 2024-02-27T06:38:33.827 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:33 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:33.486+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:33.852 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:33 smithi131 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3[80286]: 2024-02-27T06:38:33.413+0000 7fc383645700 -1 osd.3 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.225671+0000 front 2024-02-27T06:38:07.225808+0000 (oldest deadline 2024-02-27T06:38:31.325892+0000) 2024-02-27T06:38:34.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:33 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:33.819+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:34.714 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:34 smithi131 podman[113377]: 2024-02-27 06:38:34.424825315 +0000 UTC m=+5.616532336 container died 3d240baa2aedc2ebad6c8fbc11825c8e6aceae4fd792bc79347c88079be64405 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:38:34.858 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:34 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:34.404+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:05.951598+0000 front 2024-02-27T06:38:05.951697+0000 (oldest deadline 2024-02-27T06:38:31.251661+0000) 2024-02-27T06:38:34.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:34 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:34.442+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:35.103 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:34 smithi131 podman[113377]: 2024-02-27 06:38:34.9226843 +0000 UTC m=+6.114391328 container cleanup 3d240baa2aedc2ebad6c8fbc11825c8e6aceae4fd792bc79347c88079be64405 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:38:35.103 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:34 smithi131 bash[113377]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3 2024-02-27T06:38:35.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:34 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:34.851+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:35.476 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:35 smithi131 podman[113394]: 2024-02-27 06:38:35.140817466 +0000 UTC m=+0.715577604 container remove 3d240baa2aedc2ebad6c8fbc11825c8e6aceae4fd792bc79347c88079be64405 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-27T06:38:35.637 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:35 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:35.490+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:35.637 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:35 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:35.376+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:05.951598+0000 front 2024-02-27T06:38:05.951697+0000 (oldest deadline 2024-02-27T06:38:31.251661+0000) 2024-02-27T06:38:35.787 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:35 smithi131 podman[113486]: 2024-02-27 06:38:35.372829083 +0000 UTC m=+0.017785294 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:38:35.787 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:35 smithi131 podman[113486]: 2024-02-27 06:38:35.508459727 +0000 UTC m=+0.153416014 container create 3d4d85e786ac13994307ccd8d0502e7e43c7c0969ed305ba991e4534a2a32330 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:38:35.787 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:35 smithi131 podman[113486]: 2024-02-27 06:38:35.778191009 +0000 UTC m=+0.423147230 container init 3d4d85e786ac13994307ccd8d0502e7e43c7c0969ed305ba991e4534a2a32330 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-02-27T06:38:35.907 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:35 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:35.630+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:36.075 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:35 smithi131 podman[113486]: 2024-02-27 06:38:35.781895808 +0000 UTC m=+0.426852029 container start 3d4d85e786ac13994307ccd8d0502e7e43c7c0969ed305ba991e4534a2a32330 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-02-27T06:38:36.075 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:35 smithi131 podman[113486]: 2024-02-27 06:38:35.861277255 +0000 UTC m=+0.506233467 container attach 3d4d85e786ac13994307ccd8d0502e7e43c7c0969ed305ba991e4534a2a32330 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:38:36.328 INFO:journalctl@ceph.osd.3.smithi131.stdout:Feb 27 06:38:36 smithi131 podman[113486]: 2024-02-27 06:38:36.068492602 +0000 UTC m=+0.713448813 container died 3d4d85e786ac13994307ccd8d0502e7e43c7c0969ed305ba991e4534a2a32330 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:38:36.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:35 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:35.900+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:36.680 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.3.service' 2024-02-27T06:38:36.722 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T06:38:36.722 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-27T06:38:36.722 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-27T06:38:36.722 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.4 2024-02-27T06:38:36.754 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:36 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:36.382+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:05.951598+0000 front 2024-02-27T06:38:05.951697+0000 (oldest deadline 2024-02-27T06:38:31.251661+0000) 2024-02-27T06:38:36.754 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:36 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:36.614+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:36.754 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:36 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:36.535+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:37.008 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:36 smithi150 systemd[1]: Stopping Ceph osd.4 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:38:37.009 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:37 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:37.001+0000 7f51ea192700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:38:37.009 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:37 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:37.001+0000 7f51ea192700 -1 osd.4 156 *** Got signal Terminated *** 2024-02-27T06:38:37.009 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:36 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:36.883+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:37.359 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:37 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:37.001+0000 7f51ea192700 -1 osd.4 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:38:37.858 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:37 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:37.371+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:05.951598+0000 front 2024-02-27T06:38:05.951697+0000 (oldest deadline 2024-02-27T06:38:31.251661+0000) 2024-02-27T06:38:37.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:37 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:37.621+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:37.859 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:37 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:37.579+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:38.357 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:37 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:37.893+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:38.608 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:38 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:38.350+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:05.951598+0000 front 2024-02-27T06:38:05.951697+0000 (oldest deadline 2024-02-27T06:38:31.251661+0000) 2024-02-27T06:38:38.608 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:38 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:38.350+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:12.952566+0000 front 2024-02-27T06:38:12.952547+0000 (oldest deadline 2024-02-27T06:38:38.252603+0000) 2024-02-27T06:38:38.872 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:38 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:38.630+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:38.872 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:38 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:38.613+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:38.872 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:38 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:38.613+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:39.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:38 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:38.864+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:39.669 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:39 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:39.387+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:05.951598+0000 front 2024-02-27T06:38:05.951697+0000 (oldest deadline 2024-02-27T06:38:31.251661+0000) 2024-02-27T06:38:39.670 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:39 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:39.387+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:12.952566+0000 front 2024-02-27T06:38:12.952547+0000 (oldest deadline 2024-02-27T06:38:38.252603+0000) 2024-02-27T06:38:39.670 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:39 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:39.601+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:39.670 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:39 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:39.601+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:40.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:39 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:39.834+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:40.108 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:39 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:39.663+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:40.856 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:40 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:40.622+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:40.856 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:40 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:40.381+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:05.951598+0000 front 2024-02-27T06:38:05.951697+0000 (oldest deadline 2024-02-27T06:38:31.251661+0000) 2024-02-27T06:38:40.856 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:40 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:40.381+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:12.952566+0000 front 2024-02-27T06:38:12.952547+0000 (oldest deadline 2024-02-27T06:38:38.252603+0000) 2024-02-27T06:38:40.857 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:40 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:40.557+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:40.857 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:40 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:40.557+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:41.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:40 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:40.849+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:41.587 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:41 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:41.357+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:05.951598+0000 front 2024-02-27T06:38:05.951697+0000 (oldest deadline 2024-02-27T06:38:31.251661+0000) 2024-02-27T06:38:41.587 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:41 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4[57702]: 2024-02-27T06:38:41.357+0000 7f51e277e700 -1 osd.4 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:12.952566+0000 front 2024-02-27T06:38:12.952547+0000 (oldest deadline 2024-02-27T06:38:38.252603+0000) 2024-02-27T06:38:41.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:41 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:41.649+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:41.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:41 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:41.649+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:15.960245+0000 front 2024-02-27T06:38:15.960230+0000 (oldest deadline 2024-02-27T06:38:40.660064+0000) 2024-02-27T06:38:41.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:41 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:41.581+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:41.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:41 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:41.581+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:42.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:41 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:41.872+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:42.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:41 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:41.872+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:42.858 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:42 smithi150 podman[78807]: 2024-02-27 06:38:42.593478411 +0000 UTC m=+5.682938243 container died 7dc53922c3097b5c188f676604f2a17945c4db584d15d1f9fd7356aed6c5c34e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS) 2024-02-27T06:38:42.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:42 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:42.663+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:42.859 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:42 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:42.663+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:15.960245+0000 front 2024-02-27T06:38:15.960230+0000 (oldest deadline 2024-02-27T06:38:40.660064+0000) 2024-02-27T06:38:42.859 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:42 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:42.542+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:42.859 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:42 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:42.542+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:43.318 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:43 smithi150 podman[78807]: 2024-02-27 06:38:43.052301849 +0000 UTC m=+6.141761682 container cleanup 7dc53922c3097b5c188f676604f2a17945c4db584d15d1f9fd7356aed6c5c34e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-02-27T06:38:43.318 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:43 smithi150 bash[78807]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4 2024-02-27T06:38:43.319 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:42 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:42.908+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:43.319 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:42 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:42.908+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:43.608 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:43 smithi150 podman[78858]: 2024-02-27 06:38:43.312096078 +0000 UTC m=+0.719265755 container remove 7dc53922c3097b5c188f676604f2a17945c4db584d15d1f9fd7356aed6c5c34e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS) 2024-02-27T06:38:43.608 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:43 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:43.499+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:43.609 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:43 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:43.499+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:43.997 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:43 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:43.902+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:43.997 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:43 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:43.902+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:43.997 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:43 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:43.690+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:43.998 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:43 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:43.690+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:15.960245+0000 front 2024-02-27T06:38:15.960230+0000 (oldest deadline 2024-02-27T06:38:40.660064+0000) 2024-02-27T06:38:43.998 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:43 smithi150 podman[78948]: 2024-02-27 06:38:43.571769511 +0000 UTC m=+0.018047204 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:38:43.998 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:43 smithi150 podman[78948]: 2024-02-27 06:38:43.679777074 +0000 UTC m=+0.126054743 container create 4a322a4c63a4ac75c29130db466da8c17afd321bce61661e981b005224679f7e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-02-27T06:38:43.998 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:43 smithi150 podman[78948]: 2024-02-27 06:38:43.899249442 +0000 UTC m=+0.345527113 container init 4a322a4c63a4ac75c29130db466da8c17afd321bce61661e981b005224679f7e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-27T06:38:43.998 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:43 smithi150 podman[78948]: 2024-02-27 06:38:43.902297392 +0000 UTC m=+0.348575063 container start 4a322a4c63a4ac75c29130db466da8c17afd321bce61661e981b005224679f7e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:38:43.998 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:43 smithi150 podman[78948]: 2024-02-27 06:38:43.99094502 +0000 UTC m=+0.437222705 container attach 4a322a4c63a4ac75c29130db466da8c17afd321bce61661e981b005224679f7e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-27T06:38:44.358 INFO:journalctl@ceph.osd.4.smithi150.stdout:Feb 27 06:38:44 smithi150 podman[78948]: 2024-02-27 06:38:44.197015704 +0000 UTC m=+0.643293385 container died 4a322a4c63a4ac75c29130db466da8c17afd321bce61661e981b005224679f7e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True) 2024-02-27T06:38:44.747 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:44 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:44.544+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:44.747 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:44 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:44.544+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:44.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.4.service' 2024-02-27T06:38:44.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T06:38:44.858 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-27T06:38:44.858 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-27T06:38:44.858 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.5 2024-02-27T06:38:45.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:44 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:44.935+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:45.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:44 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:44.935+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:45.108 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:44 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:44.739+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:45.108 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:44 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:44.739+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:15.960245+0000 front 2024-02-27T06:38:15.960230+0000 (oldest deadline 2024-02-27T06:38:40.660064+0000) 2024-02-27T06:38:45.108 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:44 smithi150 systemd[1]: Stopping Ceph osd.5 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:38:45.584 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:45 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:45.214+0000 7f00ac735700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:38:45.584 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:45 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:45.214+0000 7f00ac735700 -1 osd.5 156 *** Got signal Terminated *** 2024-02-27T06:38:45.584 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:45 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:45.214+0000 7f00ac735700 -1 osd.5 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:38:45.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:45 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:45.741+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:45.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:45 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:45.741+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:15.960245+0000 front 2024-02-27T06:38:15.960230+0000 (oldest deadline 2024-02-27T06:38:40.660064+0000) 2024-02-27T06:38:45.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:45 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:45.577+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:45.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:45 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:45.577+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:46.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:45 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:45.958+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:46.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:45 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:45.958+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:46.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:46 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:46.731+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:46.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:46 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:46.731+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:15.960245+0000 front 2024-02-27T06:38:15.960230+0000 (oldest deadline 2024-02-27T06:38:40.660064+0000) 2024-02-27T06:38:46.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:46 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:46.582+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:46.859 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:46 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:46.582+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:47.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:47 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:46.999+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:47.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:47 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:46.999+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:47.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:47 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:47.781+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:47.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:47 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:47.781+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:15.960245+0000 front 2024-02-27T06:38:15.960230+0000 (oldest deadline 2024-02-27T06:38:40.660064+0000) 2024-02-27T06:38:47.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:47 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:47.546+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:47.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:47 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:47.546+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:48.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:47 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:47.951+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:48.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:47 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:47.951+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:48.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:48 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:48.827+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:48.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:48 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:48.827+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:15.960245+0000 front 2024-02-27T06:38:15.960230+0000 (oldest deadline 2024-02-27T06:38:40.660064+0000) 2024-02-27T06:38:48.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:48 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:48.589+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:48.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:48 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:48.589+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:49.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:48 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:48.940+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:49.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:48 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:48.940+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:49.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:49 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:49.797+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:09.059010+0000 front 2024-02-27T06:38:09.058932+0000 (oldest deadline 2024-02-27T06:38:34.859342+0000) 2024-02-27T06:38:49.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:49 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5[59967]: 2024-02-27T06:38:49.797+0000 7f00a4d21700 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:15.960245+0000 front 2024-02-27T06:38:15.960230+0000 (oldest deadline 2024-02-27T06:38:40.660064+0000) 2024-02-27T06:38:49.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:49 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:49.599+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:49.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:49 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:49.599+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:50.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:49 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:49.895+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:50.359 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:49 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:49.895+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:50.359 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:49 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:49.895+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:38:50.858 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:50 smithi150 podman[79065]: 2024-02-27 06:38:50.597861421 +0000 UTC m=+5.515528807 container died 6865822553f8db19745b1f70e1c16804d0d5ebfe57a58e311a64b5b73264b59a (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-27T06:38:50.859 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:50 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:50.581+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:50.859 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:50 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:50.581+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:50.859 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:50 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:50.581+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:23.497210+0000 front 2024-02-27T06:38:23.497166+0000 (oldest deadline 2024-02-27T06:38:50.497504+0000) 2024-02-27T06:38:51.205 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:50 smithi150 podman[79065]: 2024-02-27 06:38:50.956640475 +0000 UTC m=+5.874307909 container cleanup 6865822553f8db19745b1f70e1c16804d0d5ebfe57a58e311a64b5b73264b59a (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-02-27T06:38:51.206 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:50 smithi150 bash[79065]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5 2024-02-27T06:38:51.206 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:50 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:50.852+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:51.206 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:50 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:50.852+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:51.206 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:50 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:50.852+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:38:51.531 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:51 smithi150 podman[79081]: 2024-02-27 06:38:51.199396192 +0000 UTC m=+0.598823522 container remove 6865822553f8db19745b1f70e1c16804d0d5ebfe57a58e311a64b5b73264b59a (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-02-27T06:38:51.829 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:51 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:51.562+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:51.829 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:51 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:51.562+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:51.830 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:51 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:51.562+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:23.497210+0000 front 2024-02-27T06:38:23.497166+0000 (oldest deadline 2024-02-27T06:38:50.497504+0000) 2024-02-27T06:38:51.830 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:51 smithi150 podman[79180]: 2024-02-27 06:38:51.525414815 +0000 UTC m=+0.113666331 container create 7e78bc40de3048c5f4a42991150e63d4d4494eae028e64306f9e2227bb1f2c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:38:51.830 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:51 smithi150 podman[79180]: 2024-02-27 06:38:51.430507069 +0000 UTC m=+0.018758593 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:38:51.830 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:51 smithi150 podman[79180]: 2024-02-27 06:38:51.752888573 +0000 UTC m=+0.341140101 container init 7e78bc40de3048c5f4a42991150e63d4d4494eae028e64306f9e2227bb1f2c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:38:51.830 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:51 smithi150 podman[79180]: 2024-02-27 06:38:51.756058239 +0000 UTC m=+0.344309756 container start 7e78bc40de3048c5f4a42991150e63d4d4494eae028e64306f9e2227bb1f2c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, ceph=True) 2024-02-27T06:38:52.108 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:51 smithi150 podman[79180]: 2024-02-27 06:38:51.841148739 +0000 UTC m=+0.429400268 container attach 7e78bc40de3048c5f4a42991150e63d4d4494eae028e64306f9e2227bb1f2c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-27T06:38:52.108 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:52 smithi150 podman[79180]: 2024-02-27 06:38:52.060468343 +0000 UTC m=+0.648719869 container died 7e78bc40de3048c5f4a42991150e63d4d4494eae028e64306f9e2227bb1f2c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-27T06:38:52.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:51 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:51.822+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:52.109 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:51 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:51.822+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:52.109 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:51 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:51.822+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:38:52.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.5.service' 2024-02-27T06:38:52.618 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:52 smithi150 podman[79207]: 2024-02-27 06:38:52.564819711 +0000 UTC m=+0.499008463 container remove 7e78bc40de3048c5f4a42991150e63d4d4494eae028e64306f9e2227bb1f2c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-27T06:38:52.618 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:52 smithi150 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.5.service: Deactivated successfully. 2024-02-27T06:38:52.618 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:52 smithi150 systemd[1]: Stopped Ceph osd.5 for 3fa4c766-d537-11ee-95c3-87774f69a715. 2024-02-27T06:38:52.618 INFO:journalctl@ceph.osd.5.smithi150.stdout:Feb 27 06:38:52 smithi150 systemd[1]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.5.service: Consumed 41.617s CPU time. 2024-02-27T06:38:52.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T06:38:52.638 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-27T06:38:52.638 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-27T06:38:52.638 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.6 2024-02-27T06:38:52.958 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:52 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:52.855+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:52.958 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:52 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:52.856+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:52.958 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:52 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:52.856+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:38:52.958 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:52 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:52.612+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:52.958 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:52 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:52.612+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:52.958 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:52 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:52.612+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:23.497210+0000 front 2024-02-27T06:38:23.497166+0000 (oldest deadline 2024-02-27T06:38:50.497504+0000) 2024-02-27T06:38:52.958 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:52 smithi150 systemd[1]: Stopping Ceph osd.6 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:38:52.958 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:52 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:52.950+0000 7efd303cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:38:52.958 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:52 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:52.950+0000 7efd303cd700 -1 osd.6 156 *** Got signal Terminated *** 2024-02-27T06:38:52.959 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:52 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:52.950+0000 7efd303cd700 -1 osd.6 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:38:53.858 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:53 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:53.817+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:53.858 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:53 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:53.817+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:53.858 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:53 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:53.817+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:38:53.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:53 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:53.587+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:53.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:53 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:53.587+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:53.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:53 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:53.587+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:23.497210+0000 front 2024-02-27T06:38:23.497166+0000 (oldest deadline 2024-02-27T06:38:50.497504+0000) 2024-02-27T06:38:54.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:54 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:54.588+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:54.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:54 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:54.588+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:54.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:54 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:54.588+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:23.497210+0000 front 2024-02-27T06:38:23.497166+0000 (oldest deadline 2024-02-27T06:38:50.497504+0000) 2024-02-27T06:38:55.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:54 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:54.855+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:55.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:54 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:54.855+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:55.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:54 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:54.855+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:38:55.834 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:55 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:55.542+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:55.834 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:55 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:55.542+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:55.834 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:55 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:55.542+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:23.497210+0000 front 2024-02-27T06:38:23.497166+0000 (oldest deadline 2024-02-27T06:38:50.497504+0000) 2024-02-27T06:38:55.835 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:55 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:55.542+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6830 osd.3 since back 2024-02-27T06:38:33.799129+0000 front 2024-02-27T06:38:33.799089+0000 (oldest deadline 2024-02-27T06:38:54.898971+0000) 2024-02-27T06:38:56.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:55 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:55.827+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:56.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:55 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:55.827+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:56.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:55 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:55.827+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:38:56.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:56.587+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:56.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:56.587+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:56.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:56.587+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:23.497210+0000 front 2024-02-27T06:38:23.497166+0000 (oldest deadline 2024-02-27T06:38:50.497504+0000) 2024-02-27T06:38:56.858 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:56.587+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6830 osd.3 since back 2024-02-27T06:38:33.799129+0000 front 2024-02-27T06:38:33.799089+0000 (oldest deadline 2024-02-27T06:38:54.898971+0000) 2024-02-27T06:38:56.858 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:56.805+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:56.859 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:56.806+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:56.859 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:56 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:56.806+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:38:58.108 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:57 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:57.618+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.894240+0000 front 2024-02-27T06:38:07.894134+0000 (oldest deadline 2024-02-27T06:38:30.094613+0000) 2024-02-27T06:38:58.108 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:57 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:57.618+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.496347+0000 front 2024-02-27T06:38:16.496294+0000 (oldest deadline 2024-02-27T06:38:38.196145+0000) 2024-02-27T06:38:58.108 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:57 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:57.618+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:23.497210+0000 front 2024-02-27T06:38:23.497166+0000 (oldest deadline 2024-02-27T06:38:50.497504+0000) 2024-02-27T06:38:58.108 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:57 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6[62260]: 2024-02-27T06:38:57.618+0000 7efd2c9c1700 -1 osd.6 156 heartbeat_check: no reply from 172.21.15.131:6830 osd.3 since back 2024-02-27T06:38:33.799129+0000 front 2024-02-27T06:38:33.799089+0000 (oldest deadline 2024-02-27T06:38:54.898971+0000) 2024-02-27T06:38:58.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:57 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:57.783+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:58.109 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:57 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:57.783+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:58.109 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:57 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:57.783+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:38:58.608 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:58 smithi150 podman[79286]: 2024-02-27 06:38:58.320063652 +0000 UTC m=+5.465151560 container died 1dedb9e4ac7963fe7d1ed84c9f62077919df509000b9232966ebb19ea3aa2744 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:38:59.108 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:58 smithi150 podman[79286]: 2024-02-27 06:38:58.759127221 +0000 UTC m=+5.904215137 container cleanup 1dedb9e4ac7963fe7d1ed84c9f62077919df509000b9232966ebb19ea3aa2744 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222) 2024-02-27T06:38:59.108 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:58 smithi150 bash[79286]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6 2024-02-27T06:38:59.108 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:58 smithi150 podman[79304]: 2024-02-27 06:38:58.952160954 +0000 UTC m=+0.631309840 container remove 1dedb9e4ac7963fe7d1ed84c9f62077919df509000b9232966ebb19ea3aa2744 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, maintainer=Guillaume Abrioux ) 2024-02-27T06:38:59.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:58 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:58.768+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:59.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:58 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:58.768+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:59.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:58 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:58.768+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:38:59.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:58 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:58.768+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6830 osd.3 since back 2024-02-27T06:38:32.634690+0000 front 2024-02-27T06:38:32.634711+0000 (oldest deadline 2024-02-27T06:38:58.534566+0000) 2024-02-27T06:38:59.636 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:59 smithi150 podman[79394]: 2024-02-27 06:38:59.275154853 +0000 UTC m=+0.017795731 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:38:59.637 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:59 smithi150 podman[79394]: 2024-02-27 06:38:59.39480944 +0000 UTC m=+0.137450317 container create e621490206ced48c7d8259da040b7245f4311da6ebe41c32a2d79665b58d879e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:38:59.637 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:59 smithi150 podman[79394]: 2024-02-27 06:38:59.630715563 +0000 UTC m=+0.373356443 container init e621490206ced48c7d8259da040b7245f4311da6ebe41c32a2d79665b58d879e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-02-27T06:38:59.937 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:59 smithi150 podman[79394]: 2024-02-27 06:38:59.633833978 +0000 UTC m=+0.376474856 container start e621490206ced48c7d8259da040b7245f4311da6ebe41c32a2d79665b58d879e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68) 2024-02-27T06:38:59.937 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:59 smithi150 podman[79394]: 2024-02-27 06:38:59.714120868 +0000 UTC m=+0.456761760 container attach e621490206ced48c7d8259da040b7245f4311da6ebe41c32a2d79665b58d879e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-27T06:38:59.937 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:59 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:59.723+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:38:59.937 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:59 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:59.723+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:38:59.937 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:59 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:59.723+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:38:59.937 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:38:59 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:38:59.723+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6830 osd.3 since back 2024-02-27T06:38:32.634690+0000 front 2024-02-27T06:38:32.634711+0000 (oldest deadline 2024-02-27T06:38:58.534566+0000) 2024-02-27T06:39:00.223 INFO:journalctl@ceph.osd.6.smithi150.stdout:Feb 27 06:38:59 smithi150 podman[79394]: 2024-02-27 06:38:59.931364029 +0000 UTC m=+0.674004915 container died e621490206ced48c7d8259da040b7245f4311da6ebe41c32a2d79665b58d879e (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-6-deactivate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-27T06:39:00.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.6.service' 2024-02-27T06:39:00.590 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T06:39:00.590 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-27T06:39:00.590 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-27T06:39:00.590 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.7 2024-02-27T06:39:00.858 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:00 smithi150 systemd[1]: Stopping Ceph osd.7 for 3fa4c766-d537-11ee-95c3-87774f69a715... 2024-02-27T06:39:00.858 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:00 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:00.728+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:39:00.858 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:00 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:00.728+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:39:00.858 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:00 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:00.728+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:39:00.858 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:00 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:00.728+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6830 osd.3 since back 2024-02-27T06:38:32.634690+0000 front 2024-02-27T06:38:32.634711+0000 (oldest deadline 2024-02-27T06:38:58.534566+0000) 2024-02-27T06:39:01.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:00 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:00.912+0000 7f1e6feed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-27T06:39:01.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:00 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:00.912+0000 7f1e6feed700 -1 osd.7 156 *** Got signal Terminated *** 2024-02-27T06:39:01.358 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:00 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:00.912+0000 7f1e6feed700 -1 osd.7 156 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-27T06:39:02.074 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:01 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:01.681+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:39:02.074 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:01 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:01.681+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:39:02.074 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:01 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:01.681+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:39:02.074 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:01 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:01.681+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6830 osd.3 since back 2024-02-27T06:38:32.634690+0000 front 2024-02-27T06:38:32.634711+0000 (oldest deadline 2024-02-27T06:38:58.534566+0000) 2024-02-27T06:39:03.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:02 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:02.692+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:39:03.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:02 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:02.692+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:39:03.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:02 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:02.692+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:39:03.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:02 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:02.692+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6830 osd.3 since back 2024-02-27T06:38:32.634690+0000 front 2024-02-27T06:38:32.634711+0000 (oldest deadline 2024-02-27T06:38:58.534566+0000) 2024-02-27T06:39:04.107 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:03 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:03.711+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:39:04.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:03 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:03.711+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:39:04.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:03 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:03.711+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:39:04.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:03 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:03.711+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6830 osd.3 since back 2024-02-27T06:38:32.634690+0000 front 2024-02-27T06:38:32.634711+0000 (oldest deadline 2024-02-27T06:38:58.534566+0000) 2024-02-27T06:39:05.107 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:04 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:04.736+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:39:05.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:04 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:04.736+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:39:05.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:04 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:04.736+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:39:05.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:04 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:04.736+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6830 osd.3 since back 2024-02-27T06:38:32.634690+0000 front 2024-02-27T06:38:32.634711+0000 (oldest deadline 2024-02-27T06:38:58.534566+0000) 2024-02-27T06:39:05.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:04 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:04.736+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.150:6804 osd.4 since back 2024-02-27T06:38:40.835663+0000 front 2024-02-27T06:38:40.835546+0000 (oldest deadline 2024-02-27T06:39:03.735584+0000) 2024-02-27T06:39:06.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:05 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:05.755+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6806 osd.0 since back 2024-02-27T06:38:07.531719+0000 front 2024-02-27T06:38:07.531620+0000 (oldest deadline 2024-02-27T06:38:32.831546+0000) 2024-02-27T06:39:06.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:05 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:05.755+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6814 osd.1 since back 2024-02-27T06:38:16.932722+0000 front 2024-02-27T06:38:16.932581+0000 (oldest deadline 2024-02-27T06:38:41.032468+0000) 2024-02-27T06:39:06.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:05 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:05.755+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6822 osd.2 since back 2024-02-27T06:38:25.133453+0000 front 2024-02-27T06:38:25.133472+0000 (oldest deadline 2024-02-27T06:38:49.733758+0000) 2024-02-27T06:39:06.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:05 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:05.755+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.131:6830 osd.3 since back 2024-02-27T06:38:32.634690+0000 front 2024-02-27T06:38:32.634711+0000 (oldest deadline 2024-02-27T06:38:58.534566+0000) 2024-02-27T06:39:06.108 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:05 smithi150 ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7[64520]: 2024-02-27T06:39:05.755+0000 7f1e684d9700 -1 osd.7 156 heartbeat_check: no reply from 172.21.15.150:6804 osd.4 since back 2024-02-27T06:38:40.835663+0000 front 2024-02-27T06:38:40.835546+0000 (oldest deadline 2024-02-27T06:39:03.735584+0000) 2024-02-27T06:39:06.743 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:06 smithi150 podman[79498]: 2024-02-27 06:39:06.469302896 +0000 UTC m=+5.664126490 container died 50595e9edf3211586d5b63ba8ce3df0782ff8fb39ccab6d4711cec1e16762fee (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-02-27T06:39:07.078 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:06 smithi150 podman[79498]: 2024-02-27 06:39:06.895781573 +0000 UTC m=+6.090605169 container cleanup 50595e9edf3211586d5b63ba8ce3df0782ff8fb39ccab6d4711cec1e16762fee (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, GIT_CLEAN=True) 2024-02-27T06:39:07.078 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:06 smithi150 bash[79498]: ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7 2024-02-27T06:39:07.357 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:07 smithi150 podman[79559]: 2024-02-27 06:39:07.072086276 +0000 UTC m=+0.603291833 container remove 50595e9edf3211586d5b63ba8ce3df0782ff8fb39ccab6d4711cec1e16762fee (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.vendor=CentOS) 2024-02-27T06:39:07.798 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:07 smithi150 podman[79649]: 2024-02-27 06:39:07.506248791 +0000 UTC m=+0.112737542 container create 0b2b499cab0f59b6f528f6d8e5728f29dfc1b4debbf71ddfab170d51b0e067e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-02-27T06:39:07.798 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:07 smithi150 podman[79649]: 2024-02-27 06:39:07.411415183 +0000 UTC m=+0.017903900 image pull c4eb0759a73adf713516fad0b4b92fa560ce139e0df0145a3609489031e7df16 quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e 2024-02-27T06:39:07.798 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:07 smithi150 podman[79649]: 2024-02-27 06:39:07.708518006 +0000 UTC m=+0.315006714 container init 0b2b499cab0f59b6f528f6d8e5728f29dfc1b4debbf71ddfab170d51b0e067e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-27T06:39:07.798 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:07 smithi150 podman[79649]: 2024-02-27 06:39:07.711561439 +0000 UTC m=+0.318050154 container start 0b2b499cab0f59b6f528f6d8e5728f29dfc1b4debbf71ddfab170d51b0e067e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, io.buildah.version=1.33.5) 2024-02-27T06:39:07.798 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:07 smithi150 podman[79649]: 2024-02-27 06:39:07.791840958 +0000 UTC m=+0.398329680 container attach 0b2b499cab0f59b6f528f6d8e5728f29dfc1b4debbf71ddfab170d51b0e067e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-deactivate, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-27T06:39:08.107 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:08 smithi150 podman[79649]: 2024-02-27 06:39:08.010991231 +0000 UTC m=+0.617479940 container died 0b2b499cab0f59b6f528f6d8e5728f29dfc1b4debbf71ddfab170d51b0e067e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-27T06:39:08.615 INFO:journalctl@ceph.osd.7.smithi150.stdout:Feb 27 06:39:08 smithi150 podman[79675]: 2024-02-27 06:39:08.595688601 +0000 UTC m=+0.578815818 container remove 0b2b499cab0f59b6f528f6d8e5728f29dfc1b4debbf71ddfab170d51b0e067e4 (image=quay.ceph.io/ceph-ci/ceph@sha256:ae9a097433517d420830b65e00243797dcd0062a7e391c6ac07a2472f8a1557e, name=ceph-3fa4c766-d537-11ee-95c3-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-02-26-1600-635ee68, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5) 2024-02-27T06:39:08.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fa4c766-d537-11ee-95c3-87774f69a715@osd.7.service' 2024-02-27T06:39:08.663 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-27T06:39:08.663 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-27T06:39:08.663 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 --force --keep-logs 2024-02-27T06:42:07.288 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 --force --keep-logs 2024-02-27T06:42:26.830 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-27T06:42:26.867 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-27T06:42:26.899 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-27T06:42:26.901 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/crash to /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463/remote/smithi131/crash 2024-02-27T06:42:26.902 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/crash -- . 2024-02-27T06:42:27.015 INFO:teuthology.orchestra.run.smithi131.stderr:tar: /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/crash: Cannot open: No such file or directory 2024-02-27T06:42:27.016 INFO:teuthology.orchestra.run.smithi131.stderr:tar: Error is not recoverable: exiting now 2024-02-27T06:42:27.018 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/crash to /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463/remote/smithi150/crash 2024-02-27T06:42:27.019 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/crash -- . 2024-02-27T06:42:27.063 INFO:teuthology.orchestra.run.smithi150.stderr:tar: /var/lib/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/crash: Cannot open: No such file or directory 2024-02-27T06:42:27.064 INFO:teuthology.orchestra.run.smithi150.stderr:tar: Error is not recoverable: exiting now 2024-02-27T06:42:27.064 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-27T06:42:27.065 DEBUG:teuthology.orchestra.run.smithi131:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-27T06:42:27.097 INFO:teuthology.orchestra.run.smithi131.stdout:2024-02-27T06:21:27.967462+0000 mon.smithi131 (mon.0) 743 : cluster [WRN] Health check failed: Degraded data redundancy: 1/723 objects degraded (0.138%), 1 pg degraded (PG_DEGRADED) 2024-02-27T06:42:27.098 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-27T06:42:27.098 DEBUG:teuthology.orchestra.run.smithi131:> sudo egrep '\[SEC\]' /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-27T06:42:27.169 DEBUG:teuthology.orchestra.run.smithi131:> sudo egrep '\[ERR\]' /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-27T06:42:27.240 DEBUG:teuthology.orchestra.run.smithi131:> sudo egrep '\[WRN\]' /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-27T06:42:27.311 INFO:teuthology.orchestra.run.smithi131.stdout:2024-02-27T06:21:27.967462+0000 mon.smithi131 (mon.0) 743 : cluster [WRN] Health check failed: Degraded data redundancy: 1/723 objects degraded (0.138%), 1 pg degraded (PG_DEGRADED) 2024-02-27T06:42:27.311 INFO:tasks.cephadm:Compressing logs... 2024-02-27T06:42:27.311 DEBUG:teuthology.orchestra.run.smithi131:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-27T06:42:27.353 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-27T06:42:27.379 INFO:teuthology.orchestra.run.smithi131.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-27T06:42:27.379 INFO:teuthology.orchestra.run.smithi131.stderr:: No such file or directory 2024-02-27T06:42:27.379 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mgr.smithi131.frtgqo.log 2024-02-27T06:42:27.379 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mon.smithi131.log 2024-02-27T06:42:27.379 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mds.foofs.smithi131.mhuwwb.log 2024-02-27T06:42:27.380 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mgr.smithi131.frtgqo.log: /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mon.smithi131.log: gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.audit.log 2024-02-27T06:42:27.383 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.1.log 2024-02-27T06:42:27.384 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mds.foofs.smithi131.mhuwwb.log: /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.3.log 2024-02-27T06:42:27.384 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.0.log 2024-02-27T06:42:27.384 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.cephadm.log 2024-02-27T06:42:27.386 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-27T06:42:27.386 INFO:teuthology.orchestra.run.smithi150.stderr:: No such file or directory 2024-02-27T06:42:27.386 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.6.log 2024-02-27T06:42:27.386 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mon.smithi150.log 2024-02-27T06:42:27.386 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.audit.log 2024-02-27T06:42:27.386 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mon.smithi150.log: gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mgr.smithi150.vvgbyr.log 2024-02-27T06:42:27.386 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mds.foofs.smithi150.vsnjhu.log 2024-02-27T06:42:27.386 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.audit.log: /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mgr.smithi150.vvgbyr.log: gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.4.log 2024-02-27T06:42:27.386 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mds.foofs.smithi150.vsnjhu.log: gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.7.log 2024-02-27T06:42:27.386 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.5.log 2024-02-27T06:42:27.386 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mds.foofs.smithi150.vsnjhu.log.gz 2024-02-27T06:42:27.386 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.cephadm.log 2024-02-27T06:42:27.387 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.7.log: /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.5.log: 91.6% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.audit.log.gz 2024-02-27T06:42:27.387 INFO:teuthology.orchestra.run.smithi150.stderr: 90.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-27T06:42:27.387 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.0.log: 91.5% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.audit.log.gz 2024-02-27T06:42:27.397 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.log 2024-02-27T06:42:27.397 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.cephadm.log: 89.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-27T06:42:27.398 INFO:teuthology.orchestra.run.smithi131.stderr: 85.8% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.cephadm.log.gz 2024-02-27T06:42:27.403 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.log 2024-02-27T06:42:27.403 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-volume.log 2024-02-27T06:42:27.405 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.log: 85.7% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.cephadm.log.gz 2024-02-27T06:42:27.405 INFO:teuthology.orchestra.run.smithi131.stderr: 91.8% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mds.foofs.smithi131.mhuwwb.log.gz 2024-02-27T06:42:27.405 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.2.log 2024-02-27T06:42:27.405 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-volume.log 2024-02-27T06:42:27.407 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-volume.log: 87.0% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.log.gz 2024-02-27T06:42:27.408 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.2.log: /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-volume.log: 87.0% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph.log.gz 2024-02-27T06:42:27.416 INFO:teuthology.orchestra.run.smithi150.stderr: 91.4% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-volume.log.gz 2024-02-27T06:42:27.423 INFO:teuthology.orchestra.run.smithi131.stderr: 91.4% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-volume.log.gz 2024-02-27T06:42:27.444 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mgr.smithi150.vvgbyr.log.gz 2024-02-27T06:42:27.654 INFO:teuthology.orchestra.run.smithi131.stderr: 88.2% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mgr.smithi131.frtgqo.log.gz 2024-02-27T06:42:27.880 INFO:teuthology.orchestra.run.smithi150.stderr: 92.6% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mon.smithi150.log.gz 2024-02-27T06:42:28.816 INFO:teuthology.orchestra.run.smithi131.stderr: 92.0% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-mon.smithi131.log.gz 2024-02-27T06:42:40.342 INFO:teuthology.orchestra.run.smithi131.stderr: 94.5% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.2.log.gz 2024-02-27T06:42:42.852 INFO:teuthology.orchestra.run.smithi150.stderr: 93.9% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.4.log.gz 2024-02-27T06:42:44.459 INFO:teuthology.orchestra.run.smithi150.stderr: 94.2% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.6.log.gz 2024-02-27T06:42:45.360 INFO:teuthology.orchestra.run.smithi131.stderr: 94.1% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.0.log.gz 2024-02-27T06:42:46.890 INFO:teuthology.orchestra.run.smithi131.stderr: 94.2% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.1.log.gz 2024-02-27T06:42:47.021 INFO:teuthology.orchestra.run.smithi150.stderr: 94.2% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.5.log.gz 2024-02-27T06:42:47.170 INFO:teuthology.orchestra.run.smithi131.stderr: 94.1% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.3.log.gz 2024-02-27T06:42:47.172 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-02-27T06:42:47.172 INFO:teuthology.orchestra.run.smithi131.stderr:real 0m19.806s 2024-02-27T06:42:47.173 INFO:teuthology.orchestra.run.smithi131.stderr:user 1m9.160s 2024-02-27T06:42:47.173 INFO:teuthology.orchestra.run.smithi131.stderr:sys 0m2.727s 2024-02-27T06:42:52.257 INFO:teuthology.orchestra.run.smithi150.stderr: 94.1% -- replaced with /var/log/ceph/3fa4c766-d537-11ee-95c3-87774f69a715/ceph-osd.7.log.gz 2024-02-27T06:42:52.258 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-02-27T06:42:52.259 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m24.891s 2024-02-27T06:42:52.259 INFO:teuthology.orchestra.run.smithi150.stderr:user 1m14.621s 2024-02-27T06:42:52.259 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m2.905s 2024-02-27T06:42:52.259 INFO:tasks.cephadm:Archiving logs... 2024-02-27T06:42:52.259 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/log/ceph to /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463/remote/smithi131/log 2024-02-27T06:42:52.260 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-27T06:43:01.914 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463/remote/smithi150/log 2024-02-27T06:43:01.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-27T06:43:12.096 INFO:tasks.cephadm:Removing cluster... 2024-02-27T06:43:12.096 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 --force 2024-02-27T06:43:12.468 DEBUG:teuthology.orchestra.run.smithi150:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fa4c766-d537-11ee-95c3-87774f69a715 --force 2024-02-27T06:43:12.854 INFO:tasks.cephadm:Removing cephadm ... 2024-02-27T06:43:12.855 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-27T06:43:12.870 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-27T06:43:12.884 INFO:tasks.cephadm:Teardown complete 2024-02-27T06:43:12.885 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-27T06:43:12.897 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-27T06:43:12.897 DEBUG:teuthology.orchestra.run.smithi131:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-27T06:43:12.912 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-27T06:43:12.924 INFO:teuthology.orchestra.run.smithi131.stderr:bash: line 1: ntpq: command not found 2024-02-27T06:43:12.927 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-27T06:43:12.927 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2024-02-27T06:43:12.928 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +302us[ +326us] +/- 128ms 2024-02-27T06:43:12.928 INFO:teuthology.orchestra.run.smithi131.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -512us[ -478us] +/- 74ms 2024-02-27T06:43:12.928 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +407us[ +430us] +/- 76ms 2024-02-27T06:43:12.928 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T06:43:12.937 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-02-27T06:43:12.942 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-27T06:43:12.942 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-02-27T06:43:12.942 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 0 +322us[ +322us] +/- 129ms 2024-02-27T06:43:12.942 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -533us[ -508us] +/- 74ms 2024-02-27T06:43:12.942 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 +392us[ +413us] +/- 76ms 2024-02-27T06:43:12.942 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-27T06:43:12.943 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-27T06:43:12.953 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-27T06:43:12.953 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-27T06:43:12.998 DEBUG:teuthology.orchestra.run.smithi131:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-27T06:43:13.079 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-27T06:43:13.157 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-27T06:43:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T06:43:13.188 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-27T06:43:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T06:43:13.218 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-27T06:43:13.229 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-27T06:43:13.256 INFO:teuthology.task.internal:Duration was 2109.709395 seconds 2024-02-27T06:43:13.256 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-27T06:43:13.307 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-27T06:43:13.307 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-27T06:43:13.309 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-27T06:43:13.350 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-27T06:43:13.351 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-27T06:43:13.894 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-27T06:43:13.894 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2024-02-27T06:43:13.895 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-27T06:43:13.914 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-02-27T06:43:13.914 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-27T06:43:13.962 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-27T06:43:13.962 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-27T06:43:13.965 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-27T06:43:14.030 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-27T06:43:14.030 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-27T06:43:14.033 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-27T06:43:28.106 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-27T06:43:28.117 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-27T06:43:28.118 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-27T06:43:28.144 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-27T06:43:28.176 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-27T06:43:28.187 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-27T06:43:28.189 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-27T06:43:28.211 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2024-02-27T06:43:28.246 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-02-27T06:43:28.355 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-27T06:43:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T06:43:28.403 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-27T06:43:28.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-27T06:43:28.419 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-27T06:43:28.430 INFO:teuthology.task.internal:Transferring archived files... 2024-02-27T06:43:28.431 DEBUG:teuthology.misc:Transferring archived files from smithi131:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463/remote/smithi131 2024-02-27T06:43:28.431 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-27T06:43:28.507 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463/remote/smithi150 2024-02-27T06:43:28.507 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-27T06:43:28.552 INFO:teuthology.task.internal:Removing archive directory... 2024-02-27T06:43:28.552 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-27T06:43:28.554 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-27T06:43:28.604 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-27T06:43:28.615 INFO:teuthology.task.internal:Not uploading archives. 2024-02-27T06:43:28.616 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-27T06:43:28.647 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-27T06:43:28.647 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-27T06:43:28.649 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-27T06:43:28.661 INFO:teuthology.orchestra.run.smithi131.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 27 06:43 /home/ubuntu/cephtest 2024-02-27T06:43:28.665 INFO:teuthology.orchestra.run.smithi150.stdout: 265294 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 27 06:43 /home/ubuntu/cephtest 2024-02-27T06:43:28.667 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-27T06:43:28.696 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-27T06:43:28.797 INFO:teuthology.nuke:Checking targets against current locks 2024-02-27T06:43:28.825 DEBUG:teuthology.nuke:shortname: smithi131 2024-02-27T06:43:28.825 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-27T06:43:28.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-27 05:55:55.417639', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLyMMU2WmPTrG0nEyQ2NmvUMXtb5sj/oZhk6zkIj6rVET3wh7KaevTVGclb2I20iC0f0mf/os1Vdr5fuzKhXUrU='} 2024-02-27T06:43:28.852 DEBUG:teuthology.nuke:shortname: smithi150 2024-02-27T06:43:28.852 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-27T06:43:28.876 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-02-27_05:08:20-orch:cephadm-wip-adk-testing-2024-02-26-1600-distro-default-smithi/7574463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-27 05:55:55.416283', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1EuQQJmp6qcOuc+G8GQiW/tx9dd8YBP2iomrl9Czw/K/0oLGOOuefBVJ8PZ2KxfCk/0AouJ1k7OKERdbVOv+k='} 2024-02-27T06:43:28.909 INFO:teuthology.orchestra.console.smithi131:Power off 2024-02-27T06:43:28.909 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-27T06:43:28.933 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2024-02-27T06:43:28.933 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T06:43:28.948 INFO:teuthology.orchestra.console.smithi150:Power off 2024-02-27T06:43:28.949 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-27T06:43:28.958 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-02-27T06:43:28.971 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2024-02-27T06:43:28.971 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T06:43:28.995 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-02-27T06:43:32.959 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T06:43:32.996 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T06:43:33.069 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-02-27T06:43:33.107 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-02-27T06:43:37.074 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T06:43:37.108 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T06:43:37.185 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-02-27T06:43:37.216 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-02-27T06:43:41.187 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T06:43:41.217 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-27T06:43:41.297 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-02-27T06:43:41.325 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-02-27T06:43:41.399 INFO:teuthology.orchestra.console.smithi131:Power off completed 2024-02-27T06:43:41.426 INFO:teuthology.orchestra.console.smithi150:Power off completed 2024-02-27T06:43:41.530 INFO:teuthology.run:Summary data: description: orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 2109.709395170212 failure_reason: '"2024-02-27T06:21:27.967462+0000 mon.smithi131 (mon.0) 743 : cluster [WRN] Health check failed: Degraded data redundancy: 1/723 objects degraded (0.138%), 1 pg degraded (PG_DEGRADED)" in cluster log' owner: scheduled_adking@teuthology success: false 2024-02-27T06:43:41.531 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-27T06:43:41.609 INFO:teuthology.run:FAIL